inbound_parser—a deep dive

Christopher Besch (IBM) • 25. April 2023

inbound_parser

A Deep Dive

What we'll talk about

What does the inbound_parser do?

Development History
Code Example
Wrap Up

What does the inbound_parser do?

Development History

How Small Ideas can Blow Up

Five Step Plan to Victory

Five Step Plan to Victory?

Jira

Issues
Created in agent dashboard
Jira API
Golang library: go-jira

Jira Service Management

Requests
Created in customer portal
Jira Servicedesk API
No Golang library

E-Mail Encoding

Expected

Reality

E-Mail Encoding

Charset: Windows-1252
Encoding: quoted-printable

What does that mean?

Computers work in binary
Map all characters to binary representation -> Charset (e.g. ASCII, Windows-1252, UTF-8, ...)
Email only supports 7bit ASCII

Input	A	Ü
7bit ASCII (binary)	01000001	🤷
Windows-1252 (binary)	01000001	11011100
Windows-1252 (quoted-printable)	A	=DC
Windows-1252 (quoted-printable) as 7bit ASCIII (binary)	01000001	00111101 01000100 01000011

Input	A	Ü
7bit ASCII (binary)	01000001	🤷
UTF-8 (binary)	01000001	11000011 10011100
UTF-8 (quoted-printable)	A	=C3=9C
UTF-8 (quoted-printable) as 7bit ASCIII (binary)	01000001	00111101 01000011 00110011 00111101 00111001 01000011

There's so much more

base64 encoding
HTML character references (e.g. €)
Many more charsets...

Error Handling

Recoverable Error

E.g. wrong E-Mail encoding
Fault tolerant (continue as much as possible)
-> Defensive Programming

Untolerable Error

E.g. Jira API rejects request
Fail silently -> Security
Notify Admins via mail (minimal redundant code)
-> Offensive Programming

Always

Continue receiving other E-Mails / Sendgrid Events
-> Defensive Programming

Data Logging

"Rohdaten sind geil!"—"Raw data is awesome!"
(David Kriesel)

First order of business: store raw incoming mail
Full offline testing environment

Development Workflow

Something goes wrong, (we're not built different)
Get raw data
Attempt fix, add more logging
Run test with raw data in testing environment
Solved and understood? Repeat step 3

Caveats for Deployment

GDPR compliance
Performance
Storage

Code Example

Touching Raw User Input

Find Issue Key in Subject

                            RE: DTHSERVICE-4200 Some Request

Extract Reply Message

                            [...]
                                Mit freundlichen Grüßen
                                Chris

                                PLEASE REPLY ABOVE THIS LINE

                                Dear Valued Customer,
                                [...]

Module Structure

Main Structs

Handler Function

In Action

Customer Mail

In Action

inbound_parser Log

In Action

Reply Mail

In Action

Service Request

Main Takeaways

Aim for deep understanding
Log everything
Avoid relying on raw user input
Sometimes no elegant solution