inbound_parser—a deep dive
Christopher Besch (IBM) • 25. April 2023

inbound_parser

A Deep Dive

What we'll talk about

  1. What does the inbound_parser do?
  2. Development History
  3. Code Example
  4. Wrap Up

What does the inbound_parser do?

What does the inbound_parser do?

Development History

Development History

How Small Ideas can Blow Up

Five Step Plan to Victory

Five Step Plan to Victory

Five Step Plan to Victory

Five Step Plan to Victory

Five Step Plan to Victory?

Jira
  • Issues
  • Created in agent dashboard
  • Jira API
  • Golang library: go-jira
Jira Service Management
  • Requests
  • Created in customer portal
  • Jira Servicedesk API
  • No Golang library

E-Mail Encoding

Expected
                            
                            
                        
Reality
                            
                            
                        

E-Mail Encoding

                        
                        
                    
  • Charset: Windows-1252
  • Encoding: quoted-printable

What does that mean?

  • Computers work in binary
  • Map all characters to binary representation -> Charset (e.g. ASCII, Windows-1252, UTF-8, ...)
  • Email only supports 7bit ASCII
Input A Ü
7bit ASCII (binary) 01000001 🤷
Windows-1252 (binary) 01000001 11011100
Windows-1252 (quoted-printable) A =DC
Windows-1252 (quoted-printable) as 7bit ASCIII (binary) 01000001 00111101
01000100
01000011
Input A Ü
7bit ASCII (binary) 01000001 🤷
UTF-8 (binary) 01000001 11000011
10011100
UTF-8 (quoted-printable) A =C3=9C
UTF-8 (quoted-printable) as 7bit ASCIII (binary) 01000001 00111101
01000011
00110011
00111101
00111001
01000011

There's so much more

  • base64 encoding
  • HTML character references (e.g. €)
  • Many more charsets...

Error Handling

Recoverable Error
  • E.g. wrong E-Mail encoding
  • Fault tolerant (continue as much as possible)
  • -> Defensive Programming
Untolerable Error
  • E.g. Jira API rejects request
  • Fail silently -> Security
  • Notify Admins via mail (minimal redundant code)
  • -> Offensive Programming
    Always
  • Continue receiving other E-Mails / Sendgrid Events
  • -> Defensive Programming

Data Logging

"Rohdaten sind geil!"—"Raw data is awesome!"
(David Kriesel)

  • First order of business: store raw incoming mail
  • Full offline testing environment

Development Workflow

  1. Something goes wrong, (we're not built different)
  2. Get raw data
  3. Attempt fix, add more logging
  4. Run test with raw data in testing environment
  5. Solved and understood? Repeat step 3

Caveats for Deployment

  • GDPR compliance
  • Performance
  • Storage

Code Example

Touching Raw User Input

Find Issue Key in Subject
                            
                            
                        

                            
RE: DTHSERVICE-4200 Some Request
Extract Reply Message
                            
                            
                        
                            
[...] Mit freundlichen Grüßen Chris PLEASE REPLY ABOVE THIS LINE Dear Valued Customer, [...]

Module Structure

                        
                        
                    

Main Structs

                            
                            
                        
                            
                            
                        

Handler Function

                        
                        
                    

In Action

Customer Mail

In Action

inbound_parser Log

                        
                        
                    

In Action

Reply Mail

In Action

Service Request

Main Takeaways

  1. Aim for deep understanding
  2. Log everything
  3. Avoid relying on raw user input
  4. Sometimes no elegant solution