"What should I give up?" asks Andrew Leigh, the freshman Labor parliamentarian.

"What should I give up?" he repeats, all twitchy energy. We are killing time in the green room at the Brisbane Writers Festival ahead of a session on Australian political reportage, and I have made the mistake, for want of livelier small talk, of asking the time-pressed MP whether he has launched himself onto Twitter. Though a prolific blogger, speaker and author, Leigh seems worried about the time it would take to distil his thoughts into 140 characters or less. Still, he is amenable to the idea, and explains that he is about to embark upon an online experiment. Each night, he will flip a coin to decide whether or not to tweet the following day. Then, at the close of play each evening, he will rank how micro-blogging has affected his productivity and happiness, and apply the same test to hashtag-free days. At the end of the trial, he will tally the scores, and base his final determination on simple arithmetic.

At this point I err again, by suggesting that his method seems ridiculously complicated. The former economics professor, who has the compact physique of the pre-dawn cyclist and the jug ears of an Alfred E. Neuman, shoots me a look of bewilderment. "But it's the perfect random sample," he says, in an incredulous voice.

Two contradictory thoughts enter my mind. The first is that Leigh is the next Kevin Rudd. The second is that Leigh is the next Kevin Rudd. The member for Fraser is geeky, jargonistic and has a low tolerance for those unversed in the mechanics of random samples. But he is also whip-sharp, brimming with policy ideas and clearly going places. His curriculum vitae is adorned with two graduate degrees from Harvard, a masters and a PhD. He became a professor at Australian National University at the age of 36. Nor is economics his only area of expertise. He is a product of Australia's leadership factory, the law faculty at Sydney University, whose alumni includes John Howard, Malcolm Turnbull and Tony Abbott, and he also served as an associate for the former High Court justice, Michael Kirby, a stern judge of legal talent. Leigh helped research Robert Putman's seminal study Bowling Alone, and has recently published an Australian iteration, Disconnected. Leigh, it should be noted, also worked as adviser for the former shadow trade minister, Peter Cook, but the reason why his CV is so unconventional for a modern-day MP is because his full-time political work is so limited.

Presently, he is serving out his parliamentary apprenticeship on the backbenches, where he displays the kind of pent-up energy of a prodigious schoolchild, with arm always upward, who knows every answer but is contending with a teacher who wants to give lesser pupils a chance. However, at a time when the reservoir of political talent is at drought-like levels, Leigh — or @ALeighMP, as he is now known to his followers on Twitter — has become a stand-out. He is a rain man in the midst of Canberra's Big Dry.

For our purposes, Leigh's political backstory is just as important as his biography. In 2010, he won preselection for the Canberra seat of Fraser in what, these days, is something of a rarity: a contest where factional powerbrokers did not fix the outcome. Of the eight candidates, the factional favourite was Nick Martin, the party's assistant national secretary. A one-time policy advisor to the ACT government, and an official at the Liquor Hospitality and Miscellaneous Union, his was the identikit profile of a professional politician. However, during the preselection campaign, which stretched from Australia Day to ANZAC Day, Leigh targeted individual members in their lounge rooms and branches, and treated the process almost as it were an American primary. He also calculated, in typically algebraic fashion, how he could turn the preferential voting system to his advantage. When it came to the vote, he trailed on primary votes but emerged the winner, 144 to Martin's 96, once preferences had been distributed.

With preselection came the seat. Leigh's only regret was that he had beaten George Williams, the University of New South Wales professor who was then Leigh's colleague at ANU, where Williams is a visiting fellow. Had Leigh not run, Williams, one of the country's leading constitutional law experts, would almost certainly have been the winner. Again, Leigh had done his sums. "I did feel bad about keeping George out of parliament," says Leigh. "He would have won if I had not."

Canberra is one of the few cities where such an open preselection contest could have unfolded in the Labor Party. The Australian Capital Territory does not have the same kind of factional culture as New South Wales or Victoria. Its branches have strong and independent-minded memberships. As if to underscore the point, on the same weekend that Leigh won preselection, the factions were also humbled in the neighbouring seat of Canberra. There, Australian Labor Party (ALP) members selected Gai Brodtmann, a former public servant and diplomat who had run her own public relations consultancy (she is also the wife of the ABC political editor, Chris Uhlmann). In a surprise victory, she defeated the candidate of the Right faction, a staffer for the Labor minister Tanya Plibersek. Both Leigh and Brodtmann had bucked a system that not only heavily favours political professionals, apparatchiks and hacks, but has come to be virtually monopolised by them.

Consider the members of Julia Gillard's cabinet, whose biographies read like a mustardy Who's Who of union officials and former Labor staffers. Simon Crean is a former president of the ACTU, as is Martin Ferguson. Greg Combet was its secretary. Penny Wong was the legal officer for the then-Liquor, Hospitality and Miscellaneous Union. Nicola Roxon was an organiser for the National Union of Workers. Tony Burke worked for the Shop, Distributive and Allied Employees' Association. Chris Evans was a state secretary of the Fire Brigade Union. Stephen Conroy was the superannuation officer at the Transport Workers' Union. Joe Ludwig hails from the Australian Workers' Union. The ubiquitous Bill Shorten, whose face seems to be everywhere, ran it.

Of those without such a strong union pedigree, most were Labor staffers. Wayne Swan served as an advisor to leaders Bill Hayden and Kim Beazley, as well as being the state secretary of the Queensland branch of the ALP. Stephen Smith advised former Prime Minister Paul Keating, and served as ALP state secretary in Western Australia. Craig Emerson served on Bob Hawke's staff. Jenny Macklin worked as a research coordinator at the Labour Resource Centre. Chris Bowen was a Labor councillor and mayor. Anthony Albanese worked for the then premier of NSW, Bob Carr, who himself was a state member of parliament for 22 years. As for Julia Gillard, she served as the former Victorian premier John Brumby's chief of staff before entering parliament. The only exception is the former frontman of the band Midnight Oil, Peter Garrett, who was parachuted into the seat of Kingsford Smith with the secrecy of a midnight raid. His stealth candidacy was fiercely opposed by senior figures at the ALP headquarters in Sussex Street, who immediately tried to scupper his candidacy precisely because he was an outsider. Garrett, who was not even a party member when first he announced his candidacy, is still seen as a something of a squatter in a Caucus Room that is increasingly becoming a closed shop.

The Liberal ministry is more diverse, partly because success in the private sector is seen as something of a prerequisite. The legal profession is strongly represented by George Brandis, Eric Abetz, Julie Bishop, Kevin Andrews and Sophie Mirabella. Peter Dutton is a former policeman. Scott Morrison headed up Tourism Australia, though he worked previously as a party functionary in New South Wales. Malcolm Turnbull, with his background in law, banking and high technology, has the sort of CV that reads like a composite of at least three separate high achievers. For all that, former political staffers occupy the two most senior seats on the front bench. The shadow treasurer, Joe Hockey, was a policy adviser for the New South Wales premier, John Fahey. Tony Abbott worked as John Hewson's spin doctor. Then there is Christopher Pyne, the shadow leader of the House. After graduating from university, he worked as a staffer for the former Howard government minister, Amanda Vanstone. Then, at the age of just 25, he became an MP. Intensely combative, his career bolsters the theory that the earlier a politician reaches Canberra the more chest-pumpingly partisan they become. Perhaps it should be called the "Pyne rule".

As for the young guns in the Liberal Party, two of the MPs spoken of as possible future leaders entered parliament steeped in Capital Circle politics. Take Kelly O'Dwyer. When she contested the Melbourne seat of Higgins, she portrayed herself as "a little Aussie battler with a strong lineage in small business." But the 35-year-old had also been a political combatant in Canberra, where she served for four years on the political staff of the former Treasurer Peter Costello. Another high-flyer, the Kooyong MP Josh Frydenberg, 41, worked for John Howard and Alexander Downer.

History shows that the party apparatus and union movement can produce leaders of high calibre. Bob Hawke and Paul Keating are obvious examples. Greg Combet is one of the Gillard ministry's more sure-footed performers. But the colonisation of parliament by party professionals has had a hugely degenerative effect. From the acid partisanship to the poison of Question Time, Canberra is giving off the stench of decay, as small, stagnant ponds are prone to when they fail to be replenished. A bush capital beautified by its freshwater lakes is also being uglified by its a septic politics. The talent pool is major reason. Now for the first time ever in Australian politics, former political apparatchiks occupy a majority of parliamentary seats.

"COULD BEN CHIFLEY WIN a Labor preselection today?" The question asked of the ALP two parliaments ago, by the former New South Wales state minister Rodney Cavalier, has even more resonance today. In his 2005 speech to the NSW Fabian Society, Cavalier ruled out any possibility that a railwayman from Bathurst could enter parliament, still less become leader, and lamented the emergence of a political class populated by union officials, parliamentary staffers and party employees. The archetypal Labor parliamentarian, he said, was "cased at university where she is achieving less than academic greatness. He will have demonstrated a willingness to follow a leader, not to step out of line … His first serious employment is with an MP. Or a Minister where she will know naught of the subject area of the Minister's portfolio. Or a union where he will not have worked in the industry covered by the union employing him."

Worried by a similar trend, senior conservatives have echoed the same concerns. In 2003, John Howard bemoaned the shrinking political "gene pool". The year before, a senior Liberal delivered a speech in Melbourne warning that declining party membership rolls were giving rise to a small political cadre. There were "fewer, less representative candidates" and "fewer, less-mainstream ideas on which to draw for policy". The speech was entitled Operators vs. Representatives, and its author was Tony Abbott.

When in the late-1970s academics started paying closer attention to the make-up of the federal parliament, one early study found that while it in no way reflected the age, gender or educational profile of the general populace it was at least occupationally representative. Then, in the 1980s, a spate of studies challenged that view by finding that farmers, shopkeepers, merchants and most professional groups, like lawyers, were over-represented. Since then, the most striking trend has been the rise of political functionaries. From 1970-1990, union officials, political staffers and state legislators occupied 21 per cent of the seats. When Narelle Miragliotta of Monash University and Wayne Errington of ANU crunched the numbers for 1991 to 2007, they found that the figure had more than doubled to 44 per cent.

Their latest research, which is about to be published, will show that for the first time more than half of parliamentarians — 52 per cent — worked as full-time political advisers, electorate officers, union officials and organisers, or state legislators at some stage prior to entering parliament. Of these 78 MPs, two-thirds are ALP. The Liberals, however, are catching up, according to Narelle Miragliotta.

Of the new MPs who entered parliament in 2010 who previously had full-time political jobs, two-thirds were Liberals. "This may suggest that the ALP disease is contagious," she says, although it could also be "an aberrant pattern."

The march towards political professionalisation can be traced back to the Whitlam years, and the decision to appoint ministerial advisors drawn from the ALP. By 1983, the number of advisors spread across the ministries had swelled to 210. By 2006 it had more than doubled to 445. Over the same period, MPs heavily bolstered their constituency offices, creating even more full-time political operatives. In the 1970s, one employee, an electorate secretary, usually sufficed. By 1985, it was two or three.

By the 2007 election, it was three or four. Over the years, on the Labor side at least, a caste system has come into effect: those who toiled away in electoral offices tend to end up on the backbenches, while those who worked in Canberra are earmarked for ministerial posts. Then there are the Brahmins among Brahmins — or careerists among careerists — the factional figures, like Bill Shorten and Stephen Conroy.

The corollary of rising professionalism has been a sharp decline in the amateur wing of politics. The rank and file of both major parties has been decimated. So much so that just 1 per cent of Australians are now involved in political parties, compared with the 5 per cent who now participate in environmental and animal welfare groups. The ALP's membership is half what it was at the end of World War II, and in recent years has gone into something nearing freefall. Between 2002 and 2010, its membership fell from 50,000 to 37,000. A decade ago, the party had 1,140 branches. Now it is close to dipping below 1,000. Branch-stacking has long been used be factional leaders to ensure their favoured candidates win pre-selection. Given the dwindling membership, branch stripping has been identified by party elders as just as much a problem.

The Liberal Party is facing the same membership crisis. Consider the state of the party in one of its traditional strongholds, Victoria. At the end of the war, it could boast 49,000 members. By 2008, it was just 13,373. Almost 90 per cent of members were over 60 years old, while just 6 per cent were under 30. This had led to the closure of 84 branches, almost a fifth of the total, and the party's efforts at recruiting new members had not been hugely successful. Less than half of the recruits who had joined the party since 2000 had renewed their membership.

Nor have the major political parties ever faced such stiff competition for talent. The Greens have attracted talented young progressives such as the West Australian Senator Scott Ludlam, who in previous generations might have gravitated towards Labor. There are online activist groups, too, for instance GetUp!, which now has more members —380,000 — than all the political parties combined. Its founders are two Australian Harvard graduates, Jeremy Heimans     and David Madden, who saw in online activism more potential than conventional political careers. Starved of new members, naturally the parties are bereft of new talent.

In urgent need of reinvigoration, political parties the world over are experimenting with reforms. US-style primaries are especially in vogue, the thinking being that public interest will be stirred if members of the community, as well as members of the party, pick candidates. When Bob Carr, the former Victorian state premier Steve Bracks and the former Labor Senator John Faulkner conducted a review of the ALP after its near defeat at the 2010 election, primaries were a key recommendation. Julia Gillard also favours them. But primaries are principally seen as a recruitment tool to entice new party members rather than attracting outside candidates. Moreover, when the primary model has been road-tested in Australia the results, on both fronts, have been mixed.

In Victoria, where the ALP held a primary in 2010, party elders eventually judged the process to have been too costly and of little benefit. They also claimed it had actually caused "disgruntlement" among local members. Narelle Miragliotta reckons this official view was skewered by an ingrained suspicion of devolving any power to the rank and file. "Such techniques undermine the authority of the party machine accustomed to exerting control over selection outcomes, especially factional chiefs," she notes. "One of its consequences is that the loyalty of candidates is likely to transfer to the external constituency that selected them, thereby undermining the authority of factional leaders and the party organization." For the political class, self-perpetuation requires self-preservation. Reformers will always struggle to compete against that logic.

New South Wales has embraced the primary system more enthusiastically, partly because its young ALP general secretary, Sam Dastyari, is an advocate. When in May the party selected its mayoral candidate for Sydney, the campaign even had a number of US-style trappings. Candidates participated in a televised debate hosted by Sky News, as well as a "tele-town hall meeting," where members of the public could ring in with questions. In the end, 4,331 people signed up to take part, just 5 per cent of the Sydney's 90,000 registered voters but a respectable enough start.

Rather than a genuine exercise in representative democracy, however, critics of the Sydney primary complained it was largely an exercise in public relations from an ailing party desperate to demonstrate a willingness to change. Given the strength of the present mayoral incumbent, the independent Clover Moore, it is hardly a prized ALP candidacy. Moreover, if the aim was to select a new breed of Labor candidate, it failed. The eventual winner, the refugee advocate Linda Scott, would likely have emerged the winner from a conventional pre-selection, given her backing from senior local figures in Sussex Street. Crucially, she had also worked as a staffer for Senator John Faulkner.

It may well be that the primary system encourages voter participation, and even boosts membership roles, but will it boost the talent pool? For political neophytes, it offers a chance to build a constituency outside the branches and union halls. But the process favours those with campaigning experience and established political contacts. This baptism can also be off-puttingly brutal. "It's hard to put up with all the 'fuck you' emails," said one of the Sydney candidates, "when you are on your own and don't have the backing of the party."

The Liberals have introduced plebiscites in Victoria and Queensland, which differ from primaries in that participation is restricted to party members. Again, the principal aim has been to replenish membership rolls rather than attract new parliamentary talent. One major benefit, however, has been to reduce the power of factional warlords, which explains why plebiscites have come up against such resistance in New South Wales. Peter Reith, the former Howard government minister who conducted the Liberal post-mortem after the last federal election, has urged their adoption nationwide, partly to see "ordinary people" take part rather than just those who are "politically-driven." Pre-selection will become more like the jury system, he predicts, with decisions taken out of the hands of party cliques.

Overall, however, Reith believes the Liberals are doing a better job at pre-selecting merit-based candidates than the ALP — a comment that obviously comes with a "he-would-say-that-wouldn't-he" health warning, but which also may be true. Even though former staffers, like O'Dwyer and Frydenberg, are proliferating, Reith is unconcerned. "If I thought we were getting duds it would be an issue," he says, "but they're very good." Besides, he believes the Australian system can survive on limited talent reserves. "You just need seven or eight good people at the federal level," he reckons, "and two or three at the state." Perhaps that is just as well.

THE NAMES attached to the vacancy arising from Senator Mark Arbib's bombshell decision to step down from politics were a reminder for Labor of the talent available when the net is cast further than the unions, electoral offices and corridors of Parliament House, and normal pre-selection procedures are temporarily set to one side. Michael Fullilove, a Rhodes scholar and a fellow both of Sydney's Lowy Institute and Washington's prestigious Brookings Institution, was high on the short-list. So, too, was Tim Harcourt, the self-styled Airport Economist, who studied at the University of Minnesota and Harvard, and worked at the Reserve Bank of Australia before becoming chief economist at Austrade.

Neither could be described as a clean-skin. Fullilove worked as a speechwriter for Paul Keating, while Harcourt was a research officer at the ACTU, a position once occupied by Bob Hawke. But nor could either be described as political careerists, who have made parliament the singular target of their ambitions. It does not require any great leap of imagination to think of Fullilove, a historian and foreign affairs expert, one day becoming Australia's chief diplomat.

Harcourt would vie with Andrew Leigh to become Treasurer, if the post were selected solely on merit. The entry of either into the upper house would have significantly raised its collective IQ. Ultimately, of course, the New South Wales Senate vacancy came to be filled by Bob Carr, a politician of unusual intellect and talent.

Senior Labor figures like Bruce Hawker, Carr's former chief of staff, believe that Senate vacancies should be used to recruit outsiders, much as modern-day British prime ministers have made use of the reformed House of Lords. "The Senate could be used to populate parliament with talented people," says Hawker, "but the unions control it." More controversial are his ideas for non-elected members of the Cabinet, again as a means of fast-tracking the capable. "The idea that cabinet members have to be elected doesn't fit with systems in place around the world," he says. But this American model would never be transplanted here. As Hawker concedes: "People tell me it's a great idea, but it's not going to happen."

Instead, Australian prime ministers have been left with a relatively small number of lawmakers upon whom to draw. From federation to World War II, there were 74 to 75 members of the House of Representatives, with just 36 Senators. Since the mid-1980s, the lower house has had 150 members, compared to 435 in the US House of Representatives and 650 in the House of Commons. Canberra presently operates in defiance of the rule that less equals more. Less is indeed less.

The separateness of the land-locked bush capital has also exacerbated the talent deficit. It has turned Canberra into a political dormitory town rather than a city to put down roots and live properly. Some wannabee politicians envy the Westminster practice where MPs usually live with their families in London during the week and return to their constituencies at weekends.

In Australia, where parliament sits more infrequently, there is an expectation that MPs will live in their constituencies, and do the Canberra commute. This has a doubly damaging effect. It acts as a strong deterrent for middle-aged people with families — often precisely the individuals with the experience and hinterland who would make accomplished lawmakers. The isolation of Canberra has also brought promising careers to a premature end. Lindsay Tanner, perhaps the most talented member of the first Gillard ministry, cited the time spent away from his family as one of the principle reasons for bringing his career to an abbreviated end.

This absence of family and friends contributes to the hot-house effect of the capital. MPs often end up sharing the same digs — the former Liberal leader Brendan Nelson used to live in Joe Hockey's converted garage — and spend most of their time completely immersed in politics. Rarely do they get to breathe fresh, untainted air.

Modern-day Canberra also has a habit of making even the highly intelligent look stupid. Craig Emerson is one of the few members of the Gillard ministry who holds a PhD, in economics from the Australian National University, not that you would have known it from the recent pre-planned karaoke number, No Whyalla Wipe-out, that turned him into a viral clown. Untuneful it may have been, but he had produced one of the most eloquent statements yet on the infantile shoutfest that Australian politics has become.

Noticeable as well is how former politicians are rehabilitating their public reputations once they have left the capital. John Hewson, the former Liberal leader, is a case in point. Nick Minchin, the former Senate leader, is another. Life outside Canberra renders them intelligent again. Like Bob Carr, perhaps some of these grey nomads should be pressed back into service.

For sure, the reservoir isn't entirely dry. Along with O'Dwyer and Frydenberg, much on the Liberal side is also expected of Paul Fletcher. The MP for Bradfield in Sydney holds an MBA from New York's Columbia University and enjoyed a successful career in the communications sector, but he also served as chief of staff for the former communications minister, Richard Alston, during the Howard government. Labor insiders point to promising young politicians including Jason Clare, the 40-year-old year old home affairs minister, who represents Paul Keating's former seat of Blaxland; Ed Husic, the first Muslim member of parliament; and Melissa Parke, the MP for Fremantle who worked as an international lawyer in string of trouble-spots from Kosovo to Gaza. But Clare worked as an advisor to Bob Carr, Husic was both a political staffer and national president of the Communications, Electrical and Plumbing Union of Australia.

After being usurped by Bob Carr, Tim Harcourt, the "Airport Economist," has been urged by party chieftains to seek pre-selection for Wentworth, Malcolm Turnbull's fiefdom in the Eastern Suburbs of Sydney. Internal polling, apparently, suggests he would do well. "They told me you're half-Jewish, pro-gay and have an Asian daughter, and that I should think about it," says Harcourt. "There was no discussion of ideas. There's too much identikit stuff at the moment."

Narelle Miragliotta, Australia's leading academic authority on this issue, has a hunch that the professionalisation of parliament might have peaked, although there is no empirical evidence yet to back this up. But the trend over the past two decades has been for the political class to perpetuate itself by preserving the status quo.

With respect for politicians at an historically low ebb, has not the time come for a rethink? Traditionally in Australia, the diversity debate has focused on getting more women into politics. Perhaps the focus should shift towards hunting down that more elusive of species: the talented outsider for whom politics is a calling rather than a career.

Is The Political Talent Pool Shrinking