What is open source
If you're not familiar with it, open source software (or books, movies, hardware, etc.) is software for which the source is freely available, and redistribution is allowed. The Open Source Initiative web site has a detailed definition and discussion.
In practices, this means that anyone can take the source, modify it to better meet their needs, and contribute those changes back to the project, or create a new project based on their changes. The latter is known as forking the project, and the new project is a fork of the old one. Because the source is visible to the users, the engineering processes are also often visible to the users - you can see the problem reports, changes to the source, etc.
The opposite of open source software is closed source, or proprietary, software. The source is not available, and the engineering processes are usually not visible to the end users.
Why you would use open source firmware
The second paragraph above contains the primary reason you'd want to use open source firmware in a transmitter: you can modify it to better meet your needs.
For instance, someone noticed that the popular Devo 7E transmitter - which the open source firmware deviationTx runs on - had unused control lines on the micro-controller it used. So they added two switches to the hardware, and support for the two switches to the software. That wouldn't have been possible if the source hadn't been available.
To compare it to models, open source firmware is like having kit, whereas proprietary source is like the all-in-one electronics boards used in micro and nano-sized aircraft.
So suppose you added a spotlight with servos to point it to your scale aircraft. If you can't control the servos, they have no purpose. With a kit, you can just plug the servos into spare channels on your receiver. If the receiver doesn't have enough channels, you can replace it with one with more channels. If you have all-in-one electronics, you may not know if you have enough channels, and even if you do, there may not be any way to access them. On the transmitter side, proprietary firmware should let you tie those channels to controls on the transmitter. Open source firmware lets you program a switch to move those channels in a search pattern.
Like the kit, open source firmware doesn't mean anything is possible. The kit may not have space for a receiver with enough channels, or the model may not be able to lift what you want to add. Your controller may not have enough memory for the modifications you want to make, or it may not be fast enough to do what you want as well as keeping up the channel communications.
Why you would trust open source firmware
If you're not familiar with open source software, you may now by wondering why you would trust your expensive model to software written by a bunch of amateur programmers in their spare time. The answer is - that's not the case.
Many - if not most - of the developers working on open source software are professional software developers. Or possibly students. Working on open source projects is not looked down on by most companies, and is an accepted way to develop new skills or learn about new software tools. In some companies - most notably if the engineers still have some say in the screening process, or the company is using open source software - contributions to open source software are a requirement.
Which brings up the spare time question. Open source is used in many commercial products - DVD players, DVR's, printers, etc. Apple's operating systems (but not the user interface code) are based on open source software, and many components are still open source. Microsoft uses open source in their operating systems. The Android phone and tablet operating systems are open source, which is why forks like Amazon Fire can be created. And there are at least two commercial transmitters based on open source firmware: the Turnigy 9XR and 9XR Pro, and the FrSky Taranis.
Professional developers work on these products as their job. In some cases, the people contributing to them do so as their job, because their employer uses that software in some capacity.
It may be hard to believe, but we enjoy programming. I'm sure the people who work for large hobby companies don't stop building and flying models just because they are now getting paid to do that. Personally, I'd rather use software written by someone was doing it as a labor of love than someone who was just trying to earn a paycheck. In at least one study, open source was shown to have better quality than closed source.
The fact that the open source projects have publicly readable problem reports means I can check them for issues with my model, and possible fixes - and if there's a patch for the software, I can apply it to my copy! This just isn't possible with proprietary software.
And finally, the authors of the open source firmware all trust all their aircraft with it.
That anyone can fork an open source project can cause a problem in that it makes lots of forks possible.
One popular open source firmware family - I'm going to call it the TH9X family, because it was initially written for the FlySky TH9X transmitter - has more forks than I can count. Many of them exist because the author wanted to experiment with radically different interfaces, but most of the popular ones - based on activity in their forums and source repositories - are very similar.
On the other hand, the deviationTx software is also very popular, but there aren't a lot of forks of it, and most of them have actually been incorporated into the main distribution.
I wrote an overview of the choices a while back.
The choices for hardware range for cheap clones of the original FlySky TH9X to relatively high end transmitters like the Walkera Devo 12S. I also wrote an overview of those. The only real change is the release of the Turnigy 9XR Pro. That has gotten mediocre reviews, whereas the Taranis has generally good reviews since this was published. The Taranis is generally considered the better buy for the price - at least once you factor in the cost of buying the things the Taranis comes with that the 9XR Pro doesn't.
The popular open source firmware choices have similar interfaces - and it's not at all like what you find on proprietary transmitters.
Closed source behavior
Every computerized closed source transmitter gave you roughly the same set of controls. The first four to six channels have their output from the specific controls, and that can't be changed. Beyond that, they are from specific controls, but you can move them around.
There are then specific controls for manipulating those channels in ways that depend on the aircraft type. You enable dual rates and/or expo on some of the channels, usually enabled by one of a few switches. You can set a throttle curve, and possibly have a throttle hold or throttle cut. On helicopters, you can set up pitch curves selected by flight mode. And so on.
You also usually get some small number of mixes which let you modify an output channel based on a control other than the primary one.
The goal is to make setting up the manufacturers receivers easy, and it generally does that. Which is why all of the proprietary transmitters I've seen do things this way. If you're aware of one that is radically different, please tell us about it in the comments!
The popular open source choices - most notably deviationTx, OpenTx (on the Taranis) and the TH9X variant on the Turnigy 9XR - all have similar interfaces.
The first difference with closed source comes from the fact that, while the radio protocols may be radically different, the actual RF module used in proprietary radios may well be the same, and the difference is in the way the bits are sent. So open source firmware may well let you choose a protocol. You might also be able to add an RF module, and the firmware will change modules depending on the protocol selection.
Second, the number of channels on a digital radio is just a matter of software, as each channel is a slot in the digital packet the transmitter sends to the receiver. Since the open source authors aren't making money selling the software, you can configure the number of channels for each model. The limits depend on the speed of the CPU and the protocol, not keeping the low-end models from taking sales from more expensive ones.
Third, this software typically either doesn't have mixes, or has many of them, depending on how you count.
Each channel is - or can be, depending on the firmware - a mix. Or several mixes. Each mix can use any input control (pretty much required to support multiple protocols), can be enabled by any switch, or none, can have it's own curve - including fixed values that ignore the input - and can be combined with the previous mixes in pretty much any way you'd like.
There are also virtual channels that aren't associated with a transmitter output. They can be used for creating values to be used in the output channels.
Channels can also be used as switches, with their value determining the state of the switch. Some versions layer in virtual switches, which can test channel values and use the value of other switches in determining their state. All of these can be used to enable or disable mixes.
This is by no means an exhaustive list of the features available in open source firmware - it's just a list of the features that are common among the popular versions.
The benefit of this model for transmitter software is that the firmware is very flexible. Some examples follow.
Since you can use a virtual switch for the throttle hold switch, you can create a sticky switch for the throttle hold, that doesn't turn off unless the throttle output is off. This means you can't accidentally spin the prop when you turn off the throttle hold.
The Blade 200SRX has a panic switch that enables auto-level and disables the cyclic controls. This can be done on the transmitter with open source firmware - at least if your aircraft has auto-level. Set the channel that controls auto-level to a value that turns it on, and use the same switch to enable the last mix for each cyclic channel to set the output value to the level mode. Or - since each output channel has a safety switch and safe value, not just the throttle - use that switch for the safety switch and set the appropriate safe value.
The downside to all this is that there's no guidance. There are no settings for elevons, you have to know now to set up the mixes. You have to know which channel controls the pitch, and set the curve on it, not via a pitch curve control.
The popular firmware is solving this in a variety of ways. Some have desktop software that looks more like the proprietary controls, and you can use it to create a model to download to your transmitter. Others have a traditional mode that looks like the proprietary firmware, and you have to enable advanced mode to get the full power of the firmware. These are still very much works in progress.
Since you're working with a radio transmitter, your government probably has some things to say about what you can and can't do.
Since the software doesn't change the RF behavior of the radio, but how the controls change the digital values in the RF signal, just changing the software shouldn't be a problem. But that could depend on whether the licensing is for the transmitter as a whole, or for the transmitter module that the transmitter is using.
Changing the protocol might also be a problem, depending on the country you're in.
Modifications that alter the RF behavior - adding a module, or changing the power output levels - almost certainly invalidate the license the transmitter was sold under.
In the US, you'll be ok so long as you keep the power under 1 watt on the 2.4GHz band and aren't planning on selling the thing, because that usage falls under the FCC Home-built transmitters clause.
Using open source firmware on your transmitter provides a lot of extra flexibility in how you control your aircraft, at the expense of making common but not simple tasks harder. It's not for everyone, but the only reason to avoid it if it's for you is that it might not be legal where you are.