The Challenge
Our production web applications run on Windows Server with IIS, behind Cloudflare, across two servers. Each server writes roughly 250–280 MB of IIS access logs every day. During traffic spikes, suspected bot attacks and application errors, the infrastructure team had to open those files by hand and piece the story together. Manual IIS log analysis was slow, and it created five problems:
- Slow investigations: correlating requests across two servers meant searching large files one at a time.
- High log volume: important events were buried inside hundreds of megabytes of daily log data.
- Limited traffic visibility: spotting unusual request volumes, repeated requests and automated traffic needed extra analysis.
- Slower issue identification: tracing HTTP response codes and suspicious request patterns to a cause took longer than it should.
- Operational overhead: repeated log collection pulled time away from proactive monitoring and security work.
Two past incidents showed the cost. One involved about 8.82 million requests to a production application in 48 hours. Another was a bot-activity incident with roughly 454,707 suspected fake unique users. At that scale, manual IIS log analysis is not a practical way to investigate traffic.
The Approach
We replaced file-by-file inspection with a centralized log analysis application. The goal was simple: collect logs automatically, process them once, store them in a structure built for fast queries, and give the team one interface for IIS log analysis and troubleshooting.
The application works alongside existing controls such as Cloudflare, server monitoring and application-level troubleshooting. It does not replace them. It makes the log data behind them searchable. The data flow is straightforward: production IIS servers, automated FTP collection, log processing, structured database storage, and an investigation interface.
Implementing the IIS Log Analysis Application
Four layers make up the build, each aimed at removing a manual step from IIS log analysis.
Automated Log Collection
Production IIS logs are retrieved from both servers over FTP every 15 minutes. Scheduled retrieval removes the repeated manual download and keeps the data close to current for IIS log analysis.
A Node.js Processing Layer
A Node.js backend processes the collected logs and serves the data to the application. Handling processing in one place means every investigation starts from the same dataset, whichever server a request hit.
Structured Storage in PostgreSQL and Supabase
Processed logs are stored in PostgreSQL through Supabase, using a segregated table schema designed so queries read less data. After this schema optimization, the relevant measured queries ran approximately 75–81% faster, which matters when IIS log analysis depends on searching millions of rows.
A React and TypeScript Interface
A React and TypeScript frontend gives the team one place to review request patterns, HTTP response codes and unusual activity across both servers. The platform is organized into eight core modules, so IIS log analysis capabilities can grow as operational needs change. It follows the same approach we bring to custom web applications.
The Results
The new IIS log analysis workflow gave the infrastructure team a stronger base for monitoring and troubleshooting:
- Faster queries: relevant database queries ran approximately 75–81% faster after schema optimization.
- Less manual work: automated collection and centralized access reduced repeated retrieval and inspection of log files.
- Better visibility: consolidated data from both servers made production request activity easier to investigate.
- More efficient troubleshooting: a structured view of request patterns and HTTP responses made IIS log analysis a repeatable task instead of manual file reading.
- Room to scale: the modular design supports new IIS log analysis features as requirements evolve.
- More proactive operations: abnormal traffic and suspicious requests can be investigated alongside existing security controls.
This is one of several infrastructure projects we have documented, and IIS log analysis is only the start. See how we handled CI/CD deployment automation with Docker and Kubernetes, our SDDC infrastructure migration, and our server OS upgrade guide, or browse all case studies.
