<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Shantanu Tomar's Blog]]></title><description><![CDATA[Shantanu Tomar's Blog]]></description><link>https://shantanutomar.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Tue, 03 Nov 2026 19:08:17 GMT</lastBuildDate><atom:link href="https://shantanutomar.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Building a Fair Workload-Aware Ticket Assignment System with a Local LLM]]></title><description><![CDATA[How I built a local LLM-powered triage system that reviews workload, assigns tickets fairly, sets due dates, and avoids AI black boxes.]]></description><link>https://shantanutomar.hashnode.dev/building-a-fair-workload-aware-ticket-assignment-system-with-a-local-llm</link><guid isPermaLink="true">https://shantanutomar.hashnode.dev/building-a-fair-workload-aware-ticket-assignment-system-with-a-local-llm</guid><category><![CDATA[#ai-tools]]></category><category><![CDATA[llm]]></category><category><![CDATA[automation]]></category><category><![CDATA[System Design]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[Shantanu Tomar]]></dc:creator><pubDate>Sat, 04 Jul 2026 17:34:58 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a48d37a1ec6ee1c52c7598e/aff82f9a-240c-42e7-8203-bc284beea63a.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Every engineering team has work that does not look expensive on paper but quietly drains a lot of attention.</p>
<p>For me, that work was ticket triage.</p>
<p>I work with a small engineering team of five developers maintaining a multi-tenant SaaS product used by many customer organizations. On an average day, we receive six to seven new items: bugs, feature requests, data requests, tech-support requests, and small enhancements.</p>
<p>The support team would create a ticket and then ping me to assign it to a developer.</p>
<p>One assignment was easy.</p>
<p>The hard part was doing this repeatedly while keeping the whole team’s workload in my head.</p>
<p>Who is already handling a priority bug? Who has a large task due tomorrow? Who has work waiting for review? Who can realistically take one more ticket today? Is this new item urgent enough to interrupt someone? What due date should I set?</p>
<p>I tried solving this manually. I maintained a small spreadsheet where I noted who was working on what and when each developer might be free.</p>
<p>But soon the spreadsheet became another system to maintain.</p>
<p>Every support ping caused a context switch. It interrupted the work I actually wanted to spend more time on: planning, product thinking, and building new features.</p>
<p>Eventually, I reached a point I am not proud of. I started ignoring some of the frequent calls and messages asking me to assign tickets.</p>
<p>That was the signal that the process needed to change.</p>
<p>Not because I needed to make faster manual decisions.</p>
<p>Because I needed to encode the decision-making system itself.</p>
<h2>The goal</h2>
<p>I did not want to build a flashy AI demo.</p>
<p>I wanted a system that could quietly run during working hours and do the boring but important work:</p>
<ul>
<li><p>review newly created tickets</p>
</li>
<li><p>understand their priority and type</p>
</li>
<li><p>look at the current workload of each developer</p>
</li>
<li><p>assign the ticket fairly</p>
</li>
<li><p>set a reasonable due date</p>
</li>
<li><p>leave an explanation behind</p>
</li>
<li><p>fail safely if anything goes wrong</p>
</li>
</ul>
<p>The system currently runs every 30 minutes between 9 AM and 6 PM, Monday to Friday, on an old laptop at home.</p>
<p>That schedule is intentional.</p>
<p>I did not need real-time assignment. I needed tickets to sit in triage for at most 30 minutes so the support team would not feel the need to ping me manually.</p>
<p>After about two weeks, the system had triaged roughly 80–100 tickets. I still review decisions from time to time, but I rarely need to manually assign tickets now. The pings and calls have almost gone silent, and the average time between ticket creation and assignment has improved dramatically.</p>
<h2>Why I did not just ask an LLM to assign tickets</h2>
<p>The obvious AI solution would be:</p>
<blockquote>
<p>Give the LLM all open tickets, all developers, all priorities, and ask it to pick an assignee.</p>
</blockquote>
<p>I deliberately did not do that.</p>
<p>There were a few reasons.</p>
<p>First, developer assignment is not just a language understanding problem. It is a scheduling and fairness problem.</p>
<p>A developer might already have 9–10 tasks assigned for the coming week. New tickets are added throughout the day. If I fed the full workload context to the LLM every 30 minutes, the prompt would become large, noisy, and unreliable.</p>
<p>Second, I wanted the system to be explainable.</p>
<p>If the automation assigned a ticket to someone, I wanted to know why. I wanted to inspect the score. I wanted to tweak the formula. I wanted to run dry-runs and compare decisions before allowing the system to make real changes.</p>
<p>That is much easier when Python makes the final decision.</p>
<p>Third, this started as a personal project to reduce my own operational load. I did not want to spend paid AI tokens on something that needed to run every 30 minutes, five days a week.</p>
<p>So I used a local LLM only where it made sense: classifying the ticket.</p>
<p>The principle became:</p>
<pre><code class="language-text">LLM: classify the ticket
Python: assign the ticket
Humans: handle ambiguity and product judgment
</code></pre>
<p>That separation made the system much more reliable.</p>
<p>The LLM handles judgment-heavy interpretation of a title and description. The deterministic code handles workload, fairness, due dates, and operational safety.</p>
<h2>The architecture</h2>
<p>At a high level, the system has three layers.</p>
<pre><code class="language-text">Layer 1: LLM classification
Layer 2: Deterministic workload and scheduling
Layer 3: Safety, auditability, and retry behavior
</code></pre>
<p>The end-to-end flow looks like this:</p>
<pre><code class="language-text">New ticket in Triage
        ↓
Read title and description
        ↓
Compute each developer's current workload
        ↓
Local LLM classifies priority/type/area/effort
        ↓
Python scores developer + due date candidates
        ↓
Assign ticket and set due date
        ↓
Post auto-triage explanation
        ↓
Move ticket to the next workflow stage
</code></pre>
<p>The local LLM classifies each new ticket into structured metadata:</p>
<pre><code class="language-json">{
  "priority": "P0 | P1 | P2 | P3",
  "type": "bug | feature",
  "area": "backend | frontend | data | payments | reports | unknown",
  "effort": "small | medium | large",
  "confidence": 0.0,
  "reasoning": "Short explanation"
}
</code></pre>
<p>The assignment logic then uses that structured output to decide who should get the ticket and when it should be due.</p>
<p>The important part is that the LLM does not assign the ticket directly.</p>
<p>It turns messy human-written ticket text into structured input. The scheduling system does the rest.</p>
<h2>Running it on an old laptop with Ollama</h2>
<p>I run this on an old laptop at home.</p>
<p>That laptop was already acting as a small home server for a few self-hosted services. So adding this automation there had very low friction.</p>
<p>I use Ollama with a small local model. The model does not need to be brilliant at everything. It only needs to be good enough at one narrow task: reading a short ticket title and description, then classifying the ticket into priority, type, area, and effort.</p>
<p>That constraint matters.</p>
<p>A small local model is not something I would trust to run a large planning process with a long context window and many moving parts. But for narrow classification with a tight prompt and structured output, it works well enough.</p>
<p>Latency is also acceptable. Since the script runs every 30 minutes, it usually processes only a few new tickets at a time. It does not need to respond instantly like a chat interface.</p>
<p>This setup also has one practical advantage: it runs even if my main computer is off, or if I am away from my desk.</p>
<p>The goal was not to build the most powerful AI system.</p>
<p>The goal was to build one that would quietly run every working day without asking for my attention.</p>
<h2>Workload is not the number of open tickets</h2>
<p>The first real design challenge was workload.</p>
<p>A naive system might count how many open tickets each developer has and assign the next ticket to the person with the lowest count.</p>
<p>That does not work.</p>
<p>Five tiny UI issues are not the same as one production-blocking bug. A developer handling a burning priority issue should not be treated as equally available as someone working on a few low-priority enhancements.</p>
<p>So the system calculates weighted workload.</p>
<p>Conceptually, each ticket contributes workload like this:</p>
<pre><code class="language-text">workload = priority_weight × effort_weight × type_multiplier
</code></pre>
<p>Priority matters because urgent work consumes more attention.</p>
<p>Effort matters because large tasks should count more than small tasks.</p>
<p>Type matters because bugs and features behave differently. Bugs usually cannot wait. Features often can.</p>
<p>For example, a high-priority large bug should significantly increase a developer’s workload score. A small low-priority enhancement should count much less.</p>
<p>This means the system does not ask:</p>
<blockquote>
<p>Who has the fewest tickets?</p>
</blockquote>
<p>It asks:</p>
<blockquote>
<p>Who has the least meaningful active workload?</p>
</blockquote>
<p>That difference is important.</p>
<p>Developers in our team handle a mix of bugs and features. If a developer has more feature-related work, the system can still assign them an urgent bug, because feature work is more interruptible than bug work.</p>
<p>This is one of the places where keeping the assignment logic outside the LLM paid off.</p>
<p>I arrived at the formula through many trial runs. Whenever an assignment felt unfair, I could inspect the score breakdown, adjust a weight, run a dry-run, and see how the decision changed.</p>
<p>The system became tunable like a scheduling algorithm, not mysterious like a prompt.</p>
<h2>Not all open work is equal</h2>
<p>Our tickets move through a Kanban-style workflow:</p>
<pre><code class="language-text">Triage → Figuring it out → In Progress → For Review → For Verification → Done
</code></pre>
<p>A ticket may be open, but that does not always mean the developer is actively working on it.</p>
<p>This became an important part of the workload model.</p>
<p>For example, once a developer has submitted a pull request, the ticket moves to <code>For Review</code>. At that point, most of the implementation work is done. There may still be changes requested, but the developer is usually not spending full active effort on it.</p>
<p>So the system counts <code>For Review</code> tickets as half workload.</p>
<p>Once the PR is merged, the ticket moves to <code>For Verification</code>. At that stage, the code is done and the ticket is waiting for QA, deployment, or closure. I do not want that ticket to count against the developer’s workload anymore.</p>
<p>So <code>For Verification</code> counts as zero workload.</p>
<p>Conceptually:</p>
<table>
<thead>
<tr>
<th>Workflow stage</th>
<th>Workload impact</th>
<th>Reason</th>
</tr>
</thead>
<tbody><tr>
<td>Active implementation</td>
<td>1.0</td>
<td>Developer is still working on it</td>
</tr>
<tr>
<td>For Review</td>
<td>0.5</td>
<td>PR is submitted, but review may request changes</td>
</tr>
<tr>
<td>For Verification</td>
<td>0.0</td>
<td>Development is done; waiting for verification or closure</td>
</tr>
</tbody></table>
<p>This small detail made the assignment logic much fairer.</p>
<p>A ticket waiting on another person should not fully block the developer who already did the work.</p>
<h2>Handling stale tasks</h2>
<p>Another real-world issue was stale work.</p>
<p>In a SaaS product used by many customer organizations, not every request has the same urgency. Some requests are logged because they need to be recorded. Some are low priority. Some do not fit the current roadmap. Sometimes the customer who requested the change stops following up.</p>
<p>But the ticket still exists, and it may still have an owner.</p>
<p>In the first version, the system counted all assigned open tickets as active workload. That caused a problem.</p>
<p>If one developer had many old assigned tasks that had not seen activity for a long time, the system treated them as overloaded. As a result, newer tickets were unfairly assigned to others.</p>
<p>That was technically consistent, but operationally wrong.</p>
<p>So I added a configurable workload lookback window. Recent assigned work counts toward workload. Future-due tasks still count even if they are older. But old likely-stale tasks are not treated as active workload forever.</p>
<p>At the same time, I did not want to completely ignore stale work. A large pile of old assigned tickets is still a signal that the person’s queue is not clean.</p>
<p>So the system applies a small capped stale-task penalty.</p>
<p>The tradeoff is:</p>
<pre><code class="language-text">Do not count stale tasks as real active work.
Do not pretend they mean nothing either.
</code></pre>
<p>This was one of the more useful lessons from building the system: workload is not a static number. It depends on recency, ownership, workflow state, and due dates.</p>
<h2>Fairness across runs</h2>
<p>The first version of the system sometimes assigned too many tickets to the same person across separate runs.</p>
<p>When I investigated, the decisions were often justifiable. That developer had a slightly lower workload score.</p>
<p>But that exposed a different problem.</p>
<p>The system does not only need to be fair. It also needs to look fair to the humans using it.</p>
<p>If the same person keeps getting assigned new tickets repeatedly, it feels unfair, even if the score says they are technically the least loaded.</p>
<p>This is especially important for a cron-based system.</p>
<p>A single run may be fair in isolation. But several fair-looking runs can become unfair across a day or week.</p>
<p>So I added recent-assignment fairness.</p>
<p>The system keeps a compact history of recent AI assignments. Developers who received AI-assigned tickets recently get a penalty in the score. Assignments today get a stronger penalty. Assignments within the current run also affect the score, so one run does not dump all new tickets onto the same person.</p>
<p>There is also a close-score fairness rule.</p>
<p>If two developers are close enough in score, the system can prefer the person who has received fewer recent AI assignments.</p>
<p>The principle is simple:</p>
<pre><code class="language-text">When two candidates are close enough, fairness should beat a tiny score difference.
</code></pre>
<p>This made the system behave better from a human perspective.</p>
<p>It still respects workload. It still respects urgency. But it avoids repeatedly choosing the same developer just because their score is slightly lower.</p>
<h2>Due dates: urgency plus capacity</h2>
<p>The system does not only assign a developer. It also assigns a due date.</p>
<p>Due dates are internal planning dates, but we treat them as deadlines. So the scheduler has to balance urgency with realistic capacity.</p>
<p>The priority model works roughly like this:</p>
<table>
<thead>
<tr>
<th>Priority</th>
<th>Scheduling behavior</th>
</tr>
</thead>
<tbody><tr>
<td>P0</td>
<td>Due today; bypasses normal capacity checks</td>
</tr>
<tr>
<td>P1</td>
<td>Due soon; urgent but respects working days</td>
</tr>
<tr>
<td>P2</td>
<td>Scheduled into available capacity slots</td>
</tr>
<tr>
<td>P3</td>
<td>Scheduled further out into available capacity slots</td>
</tr>
</tbody></table>
<p>A P0 burning issue is treated differently from everything else. It gets today’s due date and goes to the least-loaded suitable developer. It bypasses normal day-by-day capacity checks because urgent production issues cannot wait for a perfect schedule.</p>
<p>There is still a safety cap so one developer is not silently buried under unlimited same-day P0s.</p>
<p>P1 is urgent too, but less extreme. It is scheduled soon and respects working days.</p>
<p>P2 and P3 tickets go through capacity-based scheduling.</p>
<p>Instead of assuming a developer is unavailable until their latest due date, the scheduler treats each day as having capacity slots. For example, the system can allow multiple smaller bugs per developer per day, while being more conservative with feature work.</p>
<p>This matters because modern development work is not always one large task at a time. Developers may be able to handle multiple small bugs in a day, especially with AI-assisted coding tools.</p>
<p>The scheduler considers:</p>
<ul>
<li><p>current open workload</p>
</li>
<li><p>ticket priority</p>
</li>
<li><p>effort</p>
</li>
<li><p>type</p>
</li>
<li><p>target-day load</p>
</li>
<li><p>due-date delay</p>
</li>
<li><p>recent AI assignments</p>
</li>
<li><p>current-run assignments</p>
</li>
<li><p>area match</p>
</li>
<li><p>stale-task penalty</p>
</li>
</ul>
<p>The lowest score wins.</p>
<p>The final assignment is therefore not just:</p>
<blockquote>
<p>Who is free?</p>
</blockquote>
<p>It is closer to:</p>
<blockquote>
<p>Who is the fairest available person for this specific ticket, on the earliest reasonable due date?</p>
</blockquote>
<h2>Dry-runs became part of the design process</h2>
<p>Dry-run mode was essential.</p>
<p>Before allowing the script to make real changes, I used dry-runs extensively. A dry-run would show what the system would classify, who it would assign, and what due date it would choose, without actually modifying anything.</p>
<p>This helped me tune the system safely.</p>
<p>For example, if an assignment looked wrong, I could check the score breakdown:</p>
<pre><code class="language-text">open workload component
bug count component
target day load component
due date delay component
recent assignment component
stale task penalty
skill or area bonus
final score
</code></pre>
<p>That made the system debuggable.</p>
<p>Without score breakdowns, I would have been guessing. With score breakdowns, I could see exactly why the system picked someone.</p>
<p>Dry-run mode was not just for safety.</p>
<p>It became the way I designed the algorithm.</p>
<h2>Failing safely</h2>
<p>For this kind of automation, doing nothing is often safer than doing the wrong thing invisibly.</p>
<p>So the system is designed to fail safely.</p>
<p>If classification fails, the ticket remains in triage.</p>
<p>If the assignment write fails, the ticket remains in triage.</p>
<p>The next cron run can retry.</p>
<p>The system also has fallback behavior for classification. If the local LLM returns malformed output or fails, a deterministic heuristic classifier can produce a conservative low-confidence classification instead of silently breaking the run.</p>
<p>When a ticket is assigned successfully, the system posts an auto-triage comment with the classification, reasoning, assignee, due date, and any instructions.</p>
<p>This visible comment matters.</p>
<p>It turns the automation from a black box into something the team can inspect.</p>
<h2>What I deliberately did not automate</h2>
<p>I currently automate assignment for bugs, tech-support tickets, data requests, small enhancements, and small features.</p>
<p>I do not want the system to own large product decisions.</p>
<p>Big features still go through me. They need proper discovery, a Problem Discovery Report, and a PRD before being assigned to someone.</p>
<p>That boundary is important.</p>
<p>The system is allowed to assign ownership.</p>
<p>It is not allowed to decide product direction.</p>
<p>This keeps the automation useful without making it dangerous. It removes repetitive triage work, but it does not replace human judgment where ambiguity, customer impact, architecture, or product direction is involved.</p>
<h2>Results after two weeks</h2>
<p>After around two weeks, the system had triaged roughly 80 tickets.</p>
<p>The biggest improvement is not just the number of tickets assigned. It is the reduction in interruptions.</p>
<p>Before this, the support team frequently pinged or called me to assign tickets. Now those pings have almost gone silent because every new ticket waits at most 30 minutes before the automation picks it up.</p>
<p>The average time between ticket creation and assignment has improved dramatically.</p>
<p>I still review decisions from time to time, and I still intervene when a ticket is escalated or when someone disagrees with the objective decision of the tool.</p>
<p>But I rarely need to manually assign routine tickets now.</p>
<p>That was the real goal.</p>
<p>Not to build an AI project.</p>
<p>To remove a recurring operational bottleneck.</p>
<h2>Lessons learned</h2>
<h3>Use AI for judgment, not control flow</h3>
<p>The most useful design decision was limiting the LLM’s role.</p>
<p>The LLM classifies ambiguous human text. Python handles repeatable decisions.</p>
<p>That made the system cheaper, faster, more explainable, and easier to tune.</p>
<h3>Fairness is a product requirement</h3>
<p>A technically correct assignment can still feel unfair if the same person keeps getting picked.</p>
<p>Fairness across time matters, especially for cron-based automations.</p>
<h3>Open does not always mean active</h3>
<p>A ticket in review is not the same as a ticket in active development.</p>
<p>A ticket waiting for verification is not the same as a ticket that still needs code.</p>
<p>A stale ticket is not the same as a current deadline.</p>
<p>Workflow state matters.</p>
<h3>Dry-runs are essential for automation that affects people</h3>
<p>If an automation assigns work to humans, you need a way to inspect its decisions before trusting it.</p>
<p>Dry-runs and score breakdowns made this system possible to tune.</p>
<h3>Small local models can be useful when the task is narrow</h3>
<p>A small local model is not good at everything.</p>
<p>But with a tight prompt, structured output, deterministic guardrails, and a narrow task, it can be good enough to create real operational value.</p>
<h2>Closing thoughts</h2>
<p>The most interesting part of this project was not the LLM.</p>
<p>The interesting part was turning an informal management process into explicit rules.</p>
<p>Before building this, “assign this ticket fairly” lived in my head. It depended on memory, intuition, interruptions, and context switching.</p>
<p>Now the process is encoded.</p>
<p>It is not perfect. It still needs human oversight. It still needs tuning. But it is consistent, inspectable, and good enough to remove a meaningful amount of daily operational load.</p>
<p>That is the kind of AI automation I find most useful: not replacing judgment entirely, but reducing the repetitive work around it.</p>
<p>A sanitized version of the source code is available on <a href="https://github.com/shantanutomarr/basecamp-ai-triage">GitHub</a></p>
]]></content:encoded></item><item><title><![CDATA[How I Upgraded a Legacy Django LMS from Python 2.7 to Django 4.2 LTS with Zero Downtime]]></title><description><![CDATA[How I upgraded a live multi-tenant LMS from Python 2.7 and Django 1.11 to Python 3.8 and Django 4.2 LTS with zero downtime.]]></description><link>https://shantanutomar.hashnode.dev/how-i-upgraded-a-legacy-django-lms-from-python-2-7-to-django-4-2-lts-with-zero-downtime</link><guid isPermaLink="true">https://shantanutomar.hashnode.dev/how-i-upgraded-a-legacy-django-lms-from-python-2-7-to-django-4-2-lts-with-zero-downtime</guid><category><![CDATA[Django]]></category><category><![CDATA[Python]]></category><category><![CDATA[Software Engineering]]></category><category><![CDATA[Web Development]]></category><category><![CDATA[Devops]]></category><category><![CDATA[legacy code]]></category><category><![CDATA[technical-debt]]></category><dc:creator><![CDATA[Shantanu Tomar]]></dc:creator><pubDate>Sat, 04 Jul 2026 10:38:28 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a48d37a1ec6ee1c52c7598e/bb11b0a7-9338-4450-a04f-31ec68898525.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Some years ago, I worked on one of the most difficult and satisfying engineering projects I had handled till then: upgrading a live multi-tenant LMS from Python 2.7 and Django 1.11 to Python 3.8 and Django 4.2 LTS.</p>
<p>This was before AI coding agents were available to help with large migrations like this. Looking back, I sometimes think the project would have been much easier today — or at least I would have had someone very patient to complain to while fixing the 47th deprecated import.</p>
<p>This was not a side project. It was not a clean greenfield application. It was a production system used by real institutes, teachers, students, admins, background workers, scheduled jobs, cache-heavy flows, and tenant-specific functionality.</p>
<p>The upgrade took a little over two and a half months from planning to completion.</p>
<p>The most interesting part was not that the final "requirements.txt" had Django 4.2 in it. The interesting part was that the platform kept running while the foundation underneath it changed piece by piece.</p>
<h2>Why we had to do it</h2>
<p>The system was already working. There was no product rewrite happening. Users were not asking for a Django upgrade. Nobody opens an LMS and says, “I wish this was running a newer framework.”</p>
<p>The real trigger was compliance.</p>
<p>One of our clients was going through a public-listing-related audit, and as part of that process, their management wanted the legacy environment upgraded. Python 2.7 and Django 1.11 were hard to justify from a compliance and long-term maintenance point of view.</p>
<p>So the goal became clear:</p>
<p>Upgrade the runtime and framework without disrupting production.</p>
<p>The visible product impact would be small. The engineering impact would be huge.</p>
<h2>The system I had to upgrade</h2>
<p>The production setup was fairly straightforward, but not luxurious.</p>
<p>We had:</p>
<ul>
<li><p>4 app servers</p>
</li>
<li><p>1 common Nginx server</p>
</li>
<li><p>1 Celery server</p>
</li>
<li><p>Redis for cache</p>
</li>
<li><p>PostgreSQL for the database</p>
</li>
<li><p>Gunicorn processes managed by Supervisor</p>
</li>
<li><p>No load balancer</p>
</li>
<li><p>No PgBouncer</p>
</li>
</ul>
<p>The application was a multi-tenant LMS, so we also had tenant-specific behavior and schema-related concerns to think about.</p>
<p>This matters because a framework upgrade in such a system is not just about making imports pass. It affects web requests, background workers, migrations, cache behavior, authentication flows, admin flows, exams, reports, videos, payments, certificates, live-class integrations, and many smaller pieces that users silently depend on every day.</p>
<p>The ideal outcome was boring production.</p>
<p>No drama. No visible outage. No heroic midnight recovery.</p>
<h2>I chose a staircase, not a jump</h2>
<p>The final target was Python 3.8 and Django 4.2 LTS.</p>
<p>But I did not try to jump directly from Django 1.11 to Django 4.2.</p>
<p>That would have been too risky. Too many APIs had changed. Too many dependencies could break at once. Too many deprecations would become hard errors together. Debugging failures would become unnecessarily difficult.</p>
<p>So I chose the slower but safer path:</p>
<pre><code class="language-plaintext">Django 1.11

→ Django 2.0
→ Django 2.1
→ Django 2.2 LTS
→ Django 3.0
→ Django 3.1
→ Django 3.2 LTS
→ Python 3.8
→ Django 4.0
→ Django 4.1
→ Django 4.2 LTS
</code></pre>
<p>Before all of this, the first major mountain was Python itself:</p>
<pre><code class="language-plaintext">Python 2.7 → Python 3.7 → Python 3.8
</code></pre>
<p>Every intermediate Django version was deployed to production slowly. There were no big jumps hidden inside local development or staging. The upgrade was done version by version, release note by release note, compatibility issue by compatibility issue.</p>
<p>It was tedious.</p>
<p>It was also the reason it worked.</p>
<h2>The first rule: strengthen the test safety net</h2>
<p>Before changing Python or Django versions, I wanted confidence that the codebase would tell me when I had broken something.</p>
<p>So the first step was not upgrading Django.</p>
<p>The first step was improving test coverage.</p>
<p>All tests had to pass before I could even think of changing versions. Where coverage was weak, new tests had to be added. I knew that once the runtime changed, failures would come from everywhere: our code, dependencies, Django internals, Python 2/3 differences, serialization, middleware, templates, management commands, and background tasks.</p>
<p>Without tests, this would have been guesswork.</p>
<p>With tests, it became a long but manageable feedback loop.</p>
<p>Along with automated tests, manual testing was also important. Critical LMS flows had to be checked before every major upgrade: login, course access, video playback, exams, payments, reports, admin workflows, Zoom/live-class flows, certificates, and other core functionality.</p>
<p>The test suite did not eliminate risk, but it made risk visible earlier.</p>
<p>That was enough to move forward.</p>
<h2>Python 2 to Python 3 was the first real battle</h2>
<p>The Python migration came before the Django migration.</p>
<p>I used "2to3", but not as a magic wand. I ran it folder by folder and reviewed the changes. The goal was not just to make the code Python 3 compatible. The goal was to make it compatible with both Python 2.7 and Python 3.7 during the transition.</p>
<p>That distinction was very important.</p>
<p>I created a separate requirements structure for the upgrade. Initially, I cloned the existing requirements and changed only the Python environment. Then I installed Python 3.7 locally, created a separate Python 3 environment, installed the new requirements, ran the local development server, and started running the test suite.</p>
<p>Many tests failed for a long time.</p>
<p>Some failures were expected. Some were surprising. Some came from our code. Some came from third-party packages. Some packages needed upgrades. Some deprecated packages had to be replaced. The requirements folder kept evolving as more incompatibilities surfaced.</p>
<p>But throughout this phase, I followed one rule strictly:</p>
<p>Every code change had to work on both Python 2.7 and Python 3.7.</p>
<h2>Backward compatibility was my rollback strategy</h2>
<p>I intentionally avoided creating a long-running upgrade branch.</p>
<p>Instead, the upgrade PRs were merged into "master", but only if they remained compatible with the old production runtime.</p>
<p>At first glance, that may sound risky. But for this migration, it was the safer path.</p>
<p>Production was still running Python 2.7. If I had kept all upgrade work in a separate branch for weeks, the eventual merge would have been enormous. It would have been hard to review, hard to test, hard to deploy, and hard to roll back.</p>
<p>By merging small backward-compatible changes into "master", production kept moving safely while the codebase gradually became Python 3 ready.</p>
<p>This made rollback much simpler.</p>
<p>If the Python 3 deployment failed, I did not need to revert a giant branch. I could switch the runtime back to Python 2.7 because the same code still supported it.</p>
<p>That idea shaped the rest of the project.</p>
<p>The upgrade was not just about reaching a new version. It was about making sure every step had an escape route.</p>
<h2>Parallel runtimes instead of replacing production in-place</h2>
<p>Our app servers used locally installed Python environments and ran the app through Gunicorn, controlled by Supervisor.</p>
<p>For the upgrade, I did not replace the existing runtime in-place.</p>
<p>I created a new Python 3.7 environment and a new Supervisor-controlled Gunicorn process that used the upgraded requirements. The old Python 2.7 Gunicorn process remained available.</p>
<p>Conceptually, it looked like this:</p>
<pre><code class="language-plaintext">Old stable runtime:
Python 2.7 + old requirements + old Gunicorn

New candidate runtime:
Python 3.7 + new requirements + new Gunicorn

Rollback:
Stop new Gunicorn → Start old Gunicorn
</code></pre>
<p>I also set up a GitHub Actions script that could switch between the new and old Gunicorn processes quickly if something went wrong.</p>
<p>This was the heart of the zero-downtime approach.</p>
<p>The new runtime was introduced as a parallel production candidate, not as a destructive replacement.</p>
<p>Later, I repeated the same pattern for Django upgrades as well: separate requirements, separate environment, separate Gunicorn process for the newer version, while keeping the last known stable runtime available for rollback.</p>
<h3>Production rollout: one slice at a time</h3>
<p>Once the unit tests and manual testing passed locally, I deployed the latest code to staging.</p>
<p>The QA and support teams tested the functionality there. Only after we were reasonably confident did I move toward production.</p>
<p>The first production step was not to upgrade all app servers.</p>
<p>Instead, we set up a new app server and routed requests from only some institutes to that server running the Python 3 setup.</p>
<p>Then we waited.</p>
<p>For two days, we watched for issues.</p>
<p>After that looked stable, we moved to the Celery server. But again, not all at once. We restarted some low-priority Celery worker queues using the new Python 3.7 environment.</p>
<p>Then we waited again.</p>
<p>After another two days without production issues, we moved all Celery workers to Python 3.7.</p>
<p>Only then did we roll out the same setup across all app servers. Even there, the Gunicorn processes were restarted one app server at a time. At any given moment, only one server was restarting.</p>
<p>This meant users did not experience meaningful downtime.</p>
<p>There was a tradeoff: during a rolling deployment, some app servers could be running the old runtime while others were running the new runtime. But because the code changes had been kept backward-compatible, that was acceptable.</p>
<p>That was the whole point of the preparation.</p>
<h2>The Redis cache bug I did not see coming</h2>
<p>One of the scariest issues appeared after upgrading to Python 3.7 and deploying to staging.</p>
<p>Redis cache data written by Python 2 could not be deserialized properly by Python 3.</p>
<p>This was a major problem because Redis was used heavily across the LMS. It was not just a harmless optimization layer. Cache touched important flows, including OTP login and other performance-sensitive areas.</p>
<p>The lesson was simple:</p>
<p>Cache is production data too.</p>
<p>The issue came from pickle compatibility. Python 2 had stored cached data using pickle protocol 2. Python 3 was trying to read data written by the old runtime and was failing because of serialization differences.</p>
<p>The fix was to make Python 3 read and write cache data in a Python-2-compatible way during the transition.</p>
<p>I set <code>PICKLE_VERSION = 2</code> in Django settings. To deserialize the old cache data correctly, "latin1" encoding had to be used because Python 2 pickles defaulted to latin1 behavior.</p>
<p>Once this was handled, Python 3 could read the Redis cache data created by Python 2.</p>
<p>This was one of those migration issues that is easy to miss in planning. The code may be compatible. The dependencies may install. The test suite may pass. But production-like data can still surprise you.</p>
<h2>After Python came the Django staircase</h2>
<p>Once Python 3 was stable in production, I repeated the same workflow for Django.</p>
<p>The Django upgrade was not one upgrade. It was a sequence of production upgrades:</p>
<pre><code class="language-plaintext">1.11 → 2.0
2.0 → 2.1
2.1 → 2.2
2.2 → 3.0
3.0 → 3.1
3.1 → 3.2
3.2 → 4.0
4.0 → 4.1
4.1 → 4.2
</code></pre>
<p>For each version, I carefully read the release notes, removed deprecated usage, upgraded or patched dependencies, ran tests, validated locally, tested on staging, and deployed slowly to production.</p>
<p>This was not glamorous work.</p>
<p>A lot of it was fixing small incompatibilities:</p>
<ul>
<li><p>removed imports</p>
</li>
<li><p>changed function signatures</p>
</li>
<li><p>deprecated Django utilities</p>
</li>
<li><p>old template tags</p>
</li>
<li><p>outdated package APIs</p>
</li>
<li><p>middleware changes</p>
</li>
<li><p>compatibility issues in third-party libraries</p>
</li>
</ul>
<p>But this is exactly why the staircase approach helped.</p>
<p>When something broke, I usually knew which step had caused it.</p>
<h2>Python 3.7 to 3.8 before Django 4.0</h2>
<p>Before moving to Django 4.0, I upgraded Python from 3.7 to 3.8 because Django 4.0 required Python 3.8 or newer.</p>
<p>Compared to Python 2.7 to Python 3.7, this was much easier. There were no massive language compatibility changes of the same kind.</p>
<p>But I still followed the same process.</p>
<p>Separate environment. Separate Gunicorn process. Tests. Staging. Production rollout. Rollback path.</p>
<p>By that point, the process had become repetitive, and that was a good thing. A boring process is exactly what you want during a risky infrastructure migration.</p>
<h2>The hidden work: third-party dependencies</h2>
<p>The hardest part of a Django upgrade is often not Django itself.</p>
<p>It is the ecosystem around Django.</p>
<p>A mature production application accumulates dependencies over years. Some remain actively maintained. Some become abandoned. Some work fine until a newer Django version removes an API they silently depended on.</p>
<p>During this upgrade, I had to deal with many dependency-related issues. Some packages could be upgraded cleanly. Some needed replacements. Some needed forks and patches.</p>
<p>A few examples:</p>
<ul>
<li><p>django-pure-pagination</p>
</li>
<li><p>django-storages</p>
</li>
<li><p>django-rest-framework-jwt</p>
</li>
<li><p>django-tenant-schemas</p>
</li>
</ul>
<p>For each package, the question was different:</p>
<p>Is there a newer version?</p>
<p>Does it support this Django version?</p>
<p>Does it still support our use case?</p>
<p>Can we replace it?</p>
<p>Do we need to patch it ourselves?</p>
<p>This dependency work is not very visible from the outside, but it consumes a large part of the real migration effort.</p>
<h2>Forking django-tenant-schemas for Django 4.2</h2>
<p>One of the most important packages in our system was "django-tenant-schemas".</p>
<p>At the time, it did not support Django 4.2 properly. Since the LMS was multi-tenant, this was not a minor dependency. It was foundational.</p>
<p>The problem appeared while running <code>migrate_schemas</code>.</p>
<p>A migration that changed the primary key field of a model from <code>AutoField</code> to <code>BigAutoField</code> failed with this error:</p>
<pre><code class="language-plaintext">NotImplementedError: subclasses of BaseDatabaseIntrospection may require a get_sequences() method
</code></pre>
<p>Django 4.2 expected the database introspection backend to provide <code>get_sequences()</code>. The tenant schemas backend did not implement it.</p>
<p>So I forked the package and added the missing support.</p>
<p>I also raised a pull request in the original open-source project. Upstream support came months later, after our production system had already been fully upgraded.</p>
<p>This was one of the moments where the upgrade stopped being just release-note work and became framework-internals work. To move forward, I had to understand what Django expected from the database backend and why the tenant schema backend was failing.</p>
<h2>Smaller patches that kept the upgrade moving</h2>
<p>There were several smaller but necessary patches too.</p>
<p>I forked and patched <code>django-pure-pagination</code> to remove the unsupported <code>ifequal</code> template tag.</p>
<p>I patched <code>django-storages</code> to deal with removed text utilities like "force_text()" and <code>smart_text()</code>.</p>
<p>I patched <code>django-rest-framework-jwt</code> to replace removed utilities such as <code>force_text</code>, <code>smart_text</code>, and <code>ugettext"</code></p>
<p>None of these issues alone would make for a dramatic migration story. But together, they represent the real nature of upgrading a legacy framework application.</p>
<p>It is rarely one big bug.</p>
<p>It is hundreds of small incompatibilities that need patience, testing, and judgment.</p>
<h2>Database migrations and tenant safety</h2>
<p>Some third-party package upgrades required Django migrations.</p>
<p>Fortunately, most of these migrations were backward-compatible. The project was not mainly a huge application-level schema redesign. The bigger challenge was making sure the upgraded framework, dependencies, and tenant schema tooling behaved correctly together.</p>
<p>The <code>migrate_schemas</code> issue with Django 4.2 was the main reminder that tenant-aware systems need extra care during framework upgrades.</p>
<p>In a single-tenant app, a migration failure is already serious. In a multi-tenant app, migration behavior has another layer of risk because the same operation may need to work correctly across tenant schemas.</p>
<h2>Observability and waiting were part of the deployment strategy</h2>
<p>During rollout, monitoring mattered a lot.</p>
<p>We had Sentry for runtime exceptions and Uptime Kuma for availability monitoring. These gave us confidence that if something broke, we would know quickly.</p>
<p>But monitoring alone was not the strategy.</p>
<p>Waiting was also part of the strategy.</p>
<p>After routing some institutes to the new Python 3 app server, we waited two days. After moving low-priority Celery queues, we waited again. We expanded the rollout only after each stage looked stable.</p>
<p>That waiting period reduced the blast radius.</p>
<p>It is tempting during an upgrade to rush once things start working. But with production systems, “it worked for five minutes” and “it has survived real usage patterns” are very different levels of confidence.</p>
<h2>The full migration timeline</h2>
<p>At a high level, the migration looked like this:</p>
<pre><code class="language-plaintext">Phase 1:
Strengthen tests and prepare separate requirements/environments.

Phase 2:
Make the codebase compatible with both Python 2.7 and Python 3.7.

Phase 3:
Deploy Python 3.7 gradually using parallel Gunicorn processes.

Phase 4:
Move Celery queues gradually.

Phase 5:
Upgrade Django version by version from 1.11 to 3.2.

Phase 6:
Upgrade Python 3.7 to 3.8.

Phase 7:
Upgrade Django 4.0 → 4.1 → 4.2 LTS.
</code></pre>
<p>The full effort took a little over two and a half months.</p>
<p>The timeline may sound long, but for a live multi-tenant LMS moving from Python 2.7 and Django 1.11 to Python 3.8 and Django 4.2 LTS, the time was justified.</p>
<p>The goal was not to finish fast.</p>
<p>The goal was to finish safely.</p>
<h2>What changed for the business</h2>
<p>From a user’s point of view, not much changed visibly.</p>
<p>That was the point.</p>
<p>Students still logged in. Teachers still used the platform. Institutes kept running their courses. Background tasks continued. The system remained available.</p>
<p>Internally, however, the platform had moved away from a legacy runtime and framework stack. The compliance concern was addressed. Future dependency upgrades became easier. Long-term maintenance risk reduced. The foundation became healthier.</p>
<p>It was one of those engineering projects where the value is mostly invisible until you understand the risk it removed.</p>
<h2>What I learned</h2>
<p>The biggest lesson I learned was that the safest upgrade is the one you can reverse quickly.</p>
<p>A few other lessons stayed with me:</p>
<p>Backward compatibility can be more valuable than clean separation during a migration.</p>
<p>Long-running upgrade branches are dangerous when production must keep moving.</p>
<p>Cache data deserves migration planning.</p>
<p>Celery workers need as much care as web servers.</p>
<p>Framework upgrades are also dependency ecosystem upgrades.</p>
<p>Reading release notes carefully saves production incidents.</p>
<p>And most importantly, zero downtime is not a final deployment trick. It is a property of the entire upgrade strategy.</p>
<p>It starts with how you structure code changes. It depends on how you test. It depends on how you manage environments. It depends on how you roll out Celery. It depends on whether rollback is simple or terrifying.</p>
<h2>The best migration is invisible</h2>
<p>Looking back, the most satisfying part of the project was not seeing Django 4.2 in the requirements file.</p>
<p>The most satisfying part was that users did not have to care.</p>
<p>The LMS kept running while the underlying Python version changed, the Django framework changed, dependencies changed, Gunicorn processes changed, Celery workers changed, and tenant-related compatibility issues were solved underneath.</p>
<p>That is what made the project meaningful.</p>
<p>A lot of engineering work is like this. If done poorly, everyone notices. If done well, almost nobody does.</p>
<p>And for this upgrade, that was exactly the outcome I wanted.</p>
]]></content:encoded></item></channel></rss>