Unfortunately the ATEM control protocol is very different, and there isn't a published API available for it.
Some clever folks have reverse engineered the ATEM protocol, and written an Arduino library for controlling the atems.
I have been wanting to port this to Q-SYS for some time, but as I don't own any BMD hardware, I haven't had the chance.
I have a few installations with ATEMs in them, and when I've had to control them, a workaround I've used is as follows:
Install an Ethernet Arduino board on the network. (I use the Ethermega from freetronics)
Write a small Arduino sketch with the parameters you need control of, and add a tcp server to the sketch to accept commands from the Q-SYS system.
In Q-SYS, write a script that speaks to the Arduino.
The ATEMS are very picky with the timing, and regularity of ACK packets, and if the timing etc gets messed up they tend to crash. One benefit of using an Arduino for this, is it offloads this burden from the core. The core should be capable of doing this if someone had the time to write a script, and access to an ATEM.
See skaarhoj.com for more info on the ATEM library.