Handling the CSV that shows up at two in the morning
Not every source has a clean API. Plenty of important data still arrives as files that a vendor or partner drops into a shared folder, an object storage bucket or an SFTP server. These feeds look simple, but they break in creative ways. A careful ingestion design saves a lot of late night fixes.
Agree on the basics in writing
Before the first file arrives, agree with the sender on the file name pattern, format, delimiter, encoding, schedule and who to contact when something goes wrong. Write it down and share it. Many file problems come from assumptions nobody stated.
Detect new files reliably
Use storage event notifications where you can, or poll on a schedule that fits the expected delivery time. Track every file you've processed by name and checksum so the same file never loads twice. Store that record somewhere durable, not in memory on one worker.
Wait until the file is complete
Large uploads can take minutes, and reading a half written file causes confusing errors. Ask the sender to upload a marker file when they're done, or to write to a temporary name and rename at the end. If neither is possible, check that the file size has stopped changing.
Keep the original untouched
Copy each file to an archive location before you parse it. If parsing logic changes or a bug turns up later, you can reload from the original without asking the vendor to resend.
Validate before loading
Check the header row, column count, row count and any required fields. Reject or quarantine files that fail instead of loading partial data. Send a clear message to the right contact about what failed and which file it was.
Handle encoding and quirks
Files from older systems may use unusual character encodings, quote rules or date formats. Set these explicitly in your parser rather than relying on defaults. Test with real sample files from the sender, not ones you made yourself. Ask for a few historical files too, since formats sometimes changed over the years.
Plan for late, missing and resent files
Alert when an expected file doesn't arrive on time. Decide how a corrected file replaces an earlier one, whether by date in the file name or by a version field. Make the load safe to rerun so a resend doesn't create duplicate rows.
Secure the transfer
Use encrypted transfer methods and give each partner its own credentials with access to only their folder. Rotate those credentials and remove access when a partnership ends.
Track it like any other pipeline
Log each file's arrival time, size, row count and load status. A simple dashboard of recent deliveries helps you spot slow drift before it becomes an outage.
A skill many teams need
File based feeds show up in finance, healthcare, retail and many other industries. To find teams doing this kind of integration work, browse Data Engineer Jobs and filter by location and work model.
The jobs page on dataengineerjobs.com.