My GSoC Journey with Django
My journey through open source and adding CompositeField to support Table-value expression in Django ORM through GSoC'26
Project at a Glance
The original goal of this project was to add support for PostgreSQL's
generate_series() function to Django's ORM. During the early discussions,
the scope became broader. Instead of supporting only one PostgreSQL function,
the goal became to build the ORM foundations needed for table-valued
expressions.
The work now has two main parts. The first is allowing a multi-column subquery
to be used as a table source. The second is using the same foundation for
set-returning functions such as generate_series(), json_each(), and
JSON_TABLE().
At the time of writing, the project code has not been merged into Django yet. The multi-column subquery work is under review, while the set-returning function work is available as a draft pull request. This report explains how the design reached its current state, what has been implemented, and what is still left to do.
My Open Source Journey Before GSoC
For the last two years, I have been actively contributing to open source, and over the past year, my involvement has become much more consistent.
I grew up in an environment where open source was almost unknown. The people around me hardly knew about the ecosystem or how contributions were even made. However, at work, we used Django and Django REST Framework (DRF). Because of that daily exposure, I became deeply interested in the framework and the open-source community behind it.
Initially, my goal was incredibly simple: I just wanted to be able to say that I had contributed to Django or DRF. At the time, getting even a single contribution merged felt like a distant dream.
One day at work, while implementing rate limiting for some endpoints, I found a possible improvement in the DRF rate-limiting API. That led me to open my first-ever pull request to one of my favorite packages. On the same day, I received a review from a contributor. The feedback seemed positive, and we discussed it for some time, but eventually, the PR was closed because it did not align with the project's policies.
At the time, it was very disappointing. Looking back, though, I realize that experience was crucial. It helped me understand how open source actually works.
After that, I shifted my focus to other Django-related projects. Fortunately, my second PR, this time to django-extensions, was merged! That experience came with its own lessons; I clearly remember accidentally rebasing the master branch onto my feature branch while trying to squash commits.
To build my confidence, I started contributing to smaller projects with around 100 to 500 stars. I went through open issues, submitted PRs, and gradually became much more comfortable with the Git workflow. Over time, I realized that contributing is not limited to fixing bugs or implementing features. You can improve documentation, update internal dependencies, remove unused code, and review other people's pull requests. Many of these high-impact contributions do not even require an open issue.
With this new understanding, I returned to DRF and other major packages. Slowly, I started seeing myself become a regular part of the community. Today, as I write this, 14 of my pull requests have been merged into Django REST Framework, and several others are currently under review. What feels even more rewarding is that I am no longer contributing to just one project. I am now a contributor to many of the most popular Django and Python packages.
Contributing to Django Core
Even though I was contributing to major community packages, contributing to Django itself still felt intimidating.
The process was very different from what I was used to. There was no simple "Issues" tab like on GitHub. Instead, I had to learn how to navigate Django's Trac ticketing system and understand the specific workflow around it.
My first two Django pull requests were actually closed. However, I kept searching through tickets and looking for ways to help. Eventually, my persistence paid off, and my PRs started getting merged. Today, I have 6 pull requests merged into Django itself. About six months before GSoC, my involvement became much more serious. I started actively participating in reviews, sharing my perspective on PRs, requesting changes, and suggesting improvements.
GSoC Participation
In the eight to nine months leading up to the GSoC application period, I was heavily involved with both Django and Wagtail. I was contributing code, reviewing PRs, and actively participating in discussions across both communities.
While reviewing a Django pull request, I came across Lily's profile. I discovered Lily's work involving Rust and Python, which immediately fascinated me because I had recently started learning Rust. I became especially interested in a project Lily maintained called django-rusty-templates.
Slowly, I started contributing to it, and eventually, I was invited to become a project collaborator. For me, this was a massive achievement. Lily is an incredibly skilled software engineer; looking at Lily's contributions to PyO3 and Django Core, I saw exactly the kind of systems-level work I had always dreamed of doing. Receiving that invitation was both exciting and overwhelming.
As GSoC approached, Django was my primary target. As soon as the project list was published, I carefully reviewed all the ideas.
The generate_series project immediately caught my attention. As someone who works almost exclusively on backend systems, I spend a lot of time communicating with databases through the ORM. The opportunity to dive deeper into how that ORM works internally felt like a rare and perfect fit. I consider myself very fortunate to have the chance to work on a project so perfectly aligned with my interests.
Writing the Proposal
Initial Exploration
The project was to add the generate_series PostgreSQL function.
Before this project, I had very little experience writing raw SQL. Most of my database work was done through Django's ORM. Because of that, I started with almost no understanding of the problem space. Fortunately, I had enough time before the proposal deadline to explore and learn.
I started researching how SELECT and FROM clauses work internally within Django's ORM. I spent a lot of time reading code, experimenting, and trying to understand the architecture.
To be completely honest, the first time I heard the name generate_series, I had no idea what it did.
So, I started researching. I read the documentation, asked questions, and learned that generate_series is a set-returning function. After that, I began experimenting with it in PostgreSQL playgrounds.
At first, I did not even know how to create a custom Django database function. But after some research and experimentation, I managed to write one and test it from the Django shell.
I wrote something like:
In the resulting SQL query, I saw:
Havinggenerate_series(1, 3) in the SELECT clause didn't make sense. It outputted the sequence, but only as a single column. I vaguely remember that only one column got printed.
I used an LLM to help me understand it better. The issue was that I couldn't reference the output of generate_series in the SELECT clause. To make it work, I needed to put it into the FROM clause, possibly using a CROSS JOIN LATERAL.
From there, another investigation started: what is the process for registering something into the FROM clause? What is the flow, and what are the criteria? In the project description, a forum post was already mentioned. On that forum, Simon discussed the potential for a generic table-valued function API. There was also a mention of FilteredRelation, which could potentially be deprecated once we had a generic API for table-valued expressions.
So, my plan was to take inspiration from the existing FilteredRelation. While exploring FilteredRelation, I looked at exactly how it registers in the FROM clause. I learned about the alias_map dictionary, which stores the table resources in the FROM clause. To live inside the alias_map, an object or relation has to possess the properties described in the JOIN class documentation.
I wrote a rough draft of my proposal with this plan. Slowly but surely, I was making progress. Once I had a clear picture, I started looking for early feedback. I remember contacting Lily, since Lily is a major contributor to the ORM. Lily is a gem of a person and is always there to help. Lily provided very critical feedback, which I took to heart to refine my proposal. When Google announced the GSoC submission dates, I submitted my proposal. I also used an LLM to help review it before submitting.
Once my proposal was submitted, I was hopeful but understandably anxious. Even though I had planned and researched as thoroughly as I could, GSoC is highly competitive, and I knew selection was far from guaranteed.
GSoC Proposal Acceptance News
As I had hoped, when I checked my phone in the morning, I saw the news that my proposal was accepted by Django. This was something I had dreamed of, but never thought would actually happen for me. For years, I had watched other people celebrate their GSoC selections. This time, it was my turn.
I went to the GSoC dashboard to see the details. I was very curious about who had been assigned as my mentor. Originally, Lily or Simon were tentatively named when Django mentioned the project. To my surprise, I saw Jacob Walls listed as my mentor.
Jacob Walls is a star in the Django community and a current Django Fellow. I had been following his work, his pull requests, and his reviews. With his decision-making skills and excellent communication, he is a true professional. I felt very lucky to have him as my mentor. He knows everything about Django, from top to bottom, and has a great reputation in the community. I was incredibly excited for the journey ahead.
Welcome by Jacob
I am very lucky to have Jacob as my mentor. The next day, he contacted me from his personal email to congratulate me. This was an overwhelming moment. When you have been following someone's work for a long time and they personally reach out to you, it feels surreal.
Project Scope Changed
Jacob followed up with another email. This one was more in-depth and covered how we would work, what we would work on, and when we would do it. He asked if anything was bugging me, like Git, my local setup, or anything else.
The most important part of the email was about shifting the scope of the project: he encouraged calling it the "Table-Valued Expression" project rather than just the "generate_series" project. The generate_series() function would be one test case for the more general ORM support.
Jacob explained that joining set-returning functions or subqueries was the most
important part of the project. Even if we only managed to get the generic
foundation right, that would still be a useful result. Users could define
their own function wrappers on top of it, while an official
generate_series() API could come later.
He suggested starting with Subquery because it gave us a cross-database way
to explore the design. Expanding JSON values into rows gave us another useful
set of test cases: PostgreSQL has jsonb_array_elements(), SQLite has
json_each(), and MySQL and Oracle have JSON_TABLE(). These examples helped
us separate the general ORM behavior from the SQL syntax of one database.
Jacob also recommended using
django-docker-box so I could run
the Django test suite against databases such as Oracle, MySQL, MariaDB,
PostgreSQL, and SQLite. We also discussed possible later work, including
FilteredRelation, cross-database JSON helpers, and lateral joins. The generic
table relation support had to come first, while these remained possible
follow-up goals.
He shared various links that were incredibly helpful for learning the basics and the initial context of the problem, including forum posts and tutorial videos by Simon on YouTube. (He also included a few other logistical details.)
How Simon's Comments and Jacob's Patch Set the Foundation
Before getting into the details of Simon's comments and hints, I have to be honest: at first, I didn't understand them.
Simon was in favor of adding a CompositeField before building the generic table relation API. Based on my research at the time, I kept wondering how a single CompositeField was going to solve the entire problem. Furthermore, writing a comprehensive model field is a massive task. It felt like a completely separate project.
Simon mentioned that there was prior work done by Lily, but it hadn't been merged because the focus had shifted toward composite primary keys. He linked Lily's pull request, and when I went through it, it was overwhelming. I didn't understand most of the code. Because I already understood my own approach, I was initially very stubborn about sticking to my original plan.
Fortunately, during our weekly meeting, Jacob cleared up all my doubts and opened my mind to new possibilities. He explained that we shouldn't get stuck on just one way of doing things; we should always be open to alternative approaches to solve a problem. He wrote a small starter patch and shared it with me. As we started iterating on it, slowly but surely, most of our problems were solved.
As I am writing this, we now support subqueries with multiple columns and multiple rows with the help of CompositeField. It is almost ready!
To give some context, here is the critical feedback from Simon that ultimately shaped our direction:
Simon's First Hint: The API and Alias
Simon shared his thoughts on how the API should look to the end-user using alias:
"As Adam alluded to, I think the bits that Django should focus on is providing the proper primitives for table-like and set-returning expressions to be inserted and referenced through the ORM. generate_series is an example of such expressions, but unnest and the more commonly requested and database-agnostic Subquery is another example of that.
If we put the problem of composite output fields aside for a moment (expressions that can return rows), I think that a good first step would be to add an Expression flag that identifies which expressions are set-returning and use Subquery as the default one to validate the approach.
With this new metadata available, we could deprecate FilteredRelation in favor of a more generic Relation expression that accepts either a resolvable relation name (like FilteredRelation does) or a set-returning expression as its first argument. It could then be used as:
Simon's Second Comment: Prioritizing CompositeField
When asked whether the focus should be purely on generate_series or on generic relation expressions, Simon emphasized the CompositeField:
"In my opinion, the main priority here would be to add generic support for CompositeField, as that would pave the door, amongst other things, to have expressions define that they return tuples of fields.
Today, there are a few cases where we fake the output_field of expressions or have to resort to manual subqueries wrapping (e.g., during aggregation), which we could do away with if we had CompositeField. Members of sql.Query.alias_map which are either BaseTable or Join could define their own output_field. This would make it easier to compose complex joins scenarios, CTEs, and implement methods such as as_subquery because the generation of the SQL associated with the FROM clause members would be delegated to the expressions instead of hardcoded in the compiler.
This would allow the lookup resolving logic of Query to be changed from starting from self.model to self.base_table.output_field and pave the door to moving tons of hardcoded Query logic to Expression.resolve_expression.
Official set-returning functions, which behave closely to subqueries, could then be built on top of that by having annotations of them result in LATERAL JOIN pushdown if they are not singletons."
Simon's Third Comment: The Path to CTE Support
Simon also highlighted how this work lays the foundation for Common Table Expressions (CTEs):
"The moment we have:
- A way to express a tuple of expressions (CompositeField) as Expression.output_field
- Adapted BaseTable and Join machinery to take advantage of 1.
The only part missing to add support for CTE is to allow compilation directives to be attached to table-like objects (e.g., a TableExpression(Expression).output_field: CompositeField that would serve as a base class for BaseTable and Join) to inform the compilation layer (compiler.SQLCompiler) that the resulting SQL should prefix the query instead of being inlined in the FROM clause."
When the First Approach Started to Break
Just when I thought things were going well, we hit another important problem.
In my implementation at the time, even a scalar subquery was being pushed into the FROM clause.
The idea behind that version was to remove a hack from Query.output_field. Because of that, even a single value or a single selected field was being wrapped in a CompositeField such as CompositeField(email=models.EmailField). Then, inside add_annotation, if the annotation looked composite, we put it into .alias_map by wrapping it in a SubqueryTable.
This turned out to be the wrong shape for the problem. I had misunderstood the .objects.alias() API and assumed that everything passed to it should go directly into the FROM clause.
To make that work, I created a SubqueryTable class as a small wrapper around the subquery. I also hardcoded the join type to CROSS JOIN. At first, that felt logical, because a cross join behaves like a nested loop and can represent a full combination of rows. But it also meant that the query could suddenly produce many more rows than expected.
A simple example showed the problem clearly:
User.objects.create(username="admin")
User.objects.create(username="public")
sub = User.objects.filter(username__isnull=False).values("username")
qs = (
User.objects
.alias(info=sub)
.annotate(the_username=F("info"))
.values("the_username")
)
# On the main branch, this does NOT do a cross join (2 results).
# On my branch, it DOES do a cross join (4 results).
print(qs)
The SQL on the main branch looked like this:
SELECT
(SELECT "U0"."username" AS "username"
FROM "auth_user" "U0"
WHERE "U0"."username" IS NOT NULL) AS "the_username"
FROM "auth_user"
LIMIT 21
But my branch produced this:
SELECT
(SELECT "U0"."username" AS "username"
FROM "auth_user" "U0"
WHERE "U0"."username" IS NOT NULL) AS "the_username"
FROM "auth_user"
CROSS JOIN
(SELECT "U0"."username" AS "username"
FROM "auth_user" "U0"
WHERE "U0"."username" IS NOT NULL) "info"
LIMIT 21
The extra info table created a cardinality explosion, so the result became n * m instead of the expected n rows.
Jacob found this problem through careful review and testing. Because of that, we decided to reduce the scope of the first pull request and focus only on multi-column subqueries for now. Even with that narrower scope, it still felt like a big step forward.
Finding a Simpler Direction
Removing the eager cross join fixed one problem, but the implementation was still more complicated than it needed to be. It was now time for more feedback from Lily and Simon, and their reviews exposed deeper problems in the design.
One of those problems was my CompositeCol class. It tried to make a
multi-column subquery work by rewriting its SELECT list. Simon pointed out
that this could change the meaning of a query that used DISTINCT. For
example, an inner query might select both email and name, but the outer
query might later use only email. Changing the selected columns at that
point could change which rows were considered distinct. The approach could
also repeat the inner subquery for every selected column, which was
inefficient.
This feedback made the main problem much clearer: I was trying to turn the
annotation into a table too early. A queryset passed to alias() should be
allowed to remain unused. It should only be added to the FROM clause if a
later queryset operation actually refers to one of its columns.
That led to a much simpler direction. The multi-column source stays in the
query's annotations at first. If filter(), values(), order_by(), or
another expression later refers to one of its fields, Django creates the
required join at that point. Once the join exists, the existing ORM machinery
can work with normal typed column references. If the same source is referenced
again, the join is reused.
The earlier implementation became difficult to improve without carrying its old design decisions forward. I therefore closed PR #21496 without merging it and started again with a cleaner implementation. Closing it did not mean that the work was wasted. Its tests, review comments, and failures helped us discover the design that followed.
Supporting Multi-Column Subqueries
The new implementation is available in PR #21636. It focuses on one foundation: using a queryset that returns multiple columns as a table source.
For example, a queryset can be stored under an alias:
user_info = User.objects.values("email", "first_name")[:1]
User.objects.alias(info=user_info).filter(
info__email="bob@example.com",
)
At first, info remains only an annotation. When info__email is referenced,
Django moves the multi-column subquery into the outer query's FROM clause
and resolves info__email to its individual column. References from
values(), filters, ordering, expressions, and tuple lookups can then use the
same join. Existing single-column subqueries keep their current behavior.
An important part of this work is CompositeField. It describes the names and
types of the columns returned by a multi-column expression. This allows a
column such as info__email to keep the normal behavior of an EmailField,
including its lookups and transforms.
The pull request is currently under review and has not been merged. Review is still shaping some details, but the lazy approach has given the implementation a much smaller and clearer foundation.
Extending the Same Idea to Set-Returning Functions
While the multi-column subquery work was being reviewed, I started applying the same approach to set-returning functions. That work is in draft PR.
The draft allows a function expression to explicitly say that it is valid as
a table source. A single-column function can use a normal Django field as its
output field. A function that returns several columns can describe them with a
CompositeField.
When one of these aliases is used, Django adds the function to the FROM
clause and gives its result a table alias. The rest of the query works with
columns from that alias instead of trying to place the function directly in
SELECT or WHERE.
The tests use real functions from different databases:
- PostgreSQL uses functions such as
generate_series()andjsonb_each(). - SQLite uses
json_each(). - MySQL, MariaDB, and Oracle use
JSON_TABLE().
The coverage has grown beyond simple examples. It now checks filtering,
selection, ordering, transforms, tuple lookups, aggregation, distinct(),
count(), exists(), combined querysets, updates, deletions, multiple table
sources, alias reuse, and AND, OR, and negated conditions. These tests have
been especially valuable because an incorrect join type can silently remove
rows or produce extra rows.
This pull request is still a draft. It depends on the multi-column subquery foundation, and its public API and internal details may continue to change during review.
Turning the Design into a DEP
As the implementation became clearer, I also wrote
a draft Django Enhancement Proposal
under Jacob's guidance. The DEP describes the intended public behavior of
table-valued expressions, how their output columns are represented, how they
are added to the FROM clause, and where database-specific behavior is still
needed.
Writing the DEP helped separate the public design from the private classes and helpers used by the current implementation. The pull request is now published for community discussion and review, and it may continue to change alongside the code reviews.
Current State and Upstream Status
At the time of writing, none of the main project code has been merged into Django. The current state is:
| Work | Status | What happened |
|---|---|---|
| PR #21496 | Closed, not merged | This was the first implementation. I closed it after its problems led us towards a cleaner approach. |
| PR #21636 | Open and under review | This adds the multi-column subquery foundation and lazy join behavior. |
| PR on my fork | Open | This extends the same foundation to set-returning and table-valued functions. |
| Table-valued expressions DEP, PR #113 | Draft and under review | I wrote this proposal under Jacob's guidance. It describes the intended behavior, API direction, database differences, and possible future work. |
The fact that the code is not merged yet is important to state clearly. ORM changes of this size need careful review because a small mistake can change the meaning or number of rows returned by an existing query. The open pull requests represent working implementations, but upstream review will decide their final shape.
What Is Left to Do
The first priority is to continue responding to review on the multi-column subquery pull request. The set-returning function branch is based on that work, so it will eventually need to be rebased onto the version that is accepted upstream.
The set-returning function draft also needs review and further simplification
before it can be considered ready. Its API names are not final yet. Full
support for correlated table sources that use OuterRef also remains future
work, together with the database-specific handling required for lateral
joins.
The draft DEP needs further discussion and review from the Django community. It should continue to describe the intended public behavior without depending too heavily on private implementation details that may change during code review.
Possible follow-up work includes official helpers for common functions such as
generate_series(), a cross-database API for expanding JSON rows, and
exploring whether the same foundation can help simplify or replace parts of
FilteredRelation.
Challenges and What I Learned
The hardest part of this project was not writing the SQL function. It was finding a design that fits naturally into Django's existing ORM without changing the behavior of existing queries.
My first approaches worked for small examples, but review revealed problems
with row counts, DISTINCT semantics, repeated subqueries, and joins created
too early. I learned that a query can produce valid SQL and still be wrong.
Testing the returned data is just as important as checking the generated SQL.
I also learned much more about how Django resolves expressions, allocates
aliases, builds its FROM clause, promotes and demotes joins, and represents
field types. Running the same feature against PostgreSQL, SQLite, MySQL,
MariaDB, and Oracle showed me how much SQL syntax and behavior can differ even
when the ORM API looks the same.
Working one failing test at a time helped me understand each part before moving to the next one. Review from Jacob, Lily, and Simon repeatedly pushed me to question my assumptions and remove abstractions that were not necessary.
The most important lesson was that starting again is not the same as losing progress. The closed pull request helped expose the real requirements. That made the later implementation smaller, more maintainable, and easier to explain.
Where the Project Stands
The project began with one PostgreSQL function, but it grew into an exploration of how Django can represent and use expressions that return complete tables. The implementation has not reached upstream yet, but we now have a clearer design, a multi-column subquery pull request under review, a cross-database set-returning function prototype, and a published draft DEP describing the wider direction.
There is still review and design work ahead. However, the project has already produced a reusable foundation, comprehensive tests, and a much better understanding of the ORM problem than we had at the beginning.
Thank You, Jacob
Before ending, I want to thank my mentor, Jacob Walls. I know he probably won't like me writing this much about him, but I have to.
Thank you, Jacob, for being so patient with me. Honestly, I gave you a lot of reasons to be frustrated. My internet disconnected in the middle of our calls, sometimes my English made it difficult for me to explain what I was thinking, and sometimes I had to reschedule our meetings. But you never made me feel bad about any of it. You always supported me and were always open when I came with a new idea.
Most importantly, your feedback helped me leave behind a complicated approach and move in a new, simpler direction. It was an honor working with you. Thank you for everything.