Category Archives: Wargames

The State of the Game

9 Replies

Grant’s Overland campaign that began at the Wilderness and ended at Petersburg. It can be argued that Grant did not win one battle but, clearly, achieved a great strategic victory. This was the beginning of the end for Lee’s Army of Northern Virginia and the beginning of the end of the American Civil War. Map by Hal Jespersen (www.posix.com/CW), found on Wikipedia. Click to enlarge.

I had just sat down to write this blog when a derecho,1)Ironically, the first person to use the Spanish word “derecho” to describe this type of storm was Gustavus D. Hinrichs, a German immigrant who settled in my hometown of Davenport in 1861. with wind speeds that peaked at 117 mph, flattened much of Iowa and dropped a hundred year old maple tree on my ’98 mustang. We also lost power and internet for a week. I spent much of that time reading old favorites including Horace Porter’s Campaigning with Grant. Porter, who served on Grant’s staff during 1864-5, provides a first person account of many extraordinary events from the start of the Overland Campaign to Lee’s surrender at Appomattox. Admittedly, the 19th century prose does get a bit flowery at times, but the he was actually there when it happened aspect of this memoir makes it invaluable.

I was struck by this passage from the beginning of the campaign and how it felt like what I wanted to write about to explain where we are on completing General Staff:

“[Grant] said,… “The only time I ever feel impatient is when I give an order for an important movement of troops in the presence of the enemy, and am waiting for them to reach their destination. Then the minutes seem like hours.” – Campaigning with Grant

At this point Grant was now General in Chief, commander of all U. S. armies including the Army of the Potomac. Orders had been issued and every U. S. army was to be on the march at this very moment and there was nothing more that Grant could do. Grant was confident that his plan would ultimately be successful. If we look at the map of the Overland campaign (above) we see that Grant wanted to fight Lee in the open. If that was not possible he would move by his left and Lee would have to respond. Eventually, Lee would be forced to entrench around Richmond and Petersburg and Grant would trap him. It was like a chess master looking sixteen plies ahead: there was nothing Lee could do, his defeat was inevitable.

There are two programmers coding General Staff:  Andy O’Neill is working on the actual game from my design documents and me (I’m working on AI).  Andy specializes in Microsoft Windows Presentation Foundation (WPF) development of business applications. General Staff is a wargaming system that utilizes multiple interlocking programs (like Microsoft Office) so it is logical to use Microsoft WPF for development. Andy is literally a  Microsoft gold medal developer. Andy also specializes in the Model-View-View Model (MVVM) development technique. This is a method commonly used in Microsoft business applications.  Despite repeated efforts to learn MVVM I confess that I am just an old coder, set in my ways, and I can’t get it. While Andy and I both write in the C# programming language, it’s best if I do not muck about with Andy’s code. What I do is write AI routines in C# and Andy imports them into his code.

In early July or late June Andy, who lives and works in Liverpool, England, fell very ill with an undefined infection. He became very dizzy and nauseous and was unable to write any code for over a month. Andy reports that he is now, “sort of…nearly…better.”

And this is why I felt like Grant on May 2, 1865: there wasn’t anything I can do to move the game forward. I’m working on the AI Engine and researching battles, maps and OOBs but that’s it. And then we lost power and the internet for a week.

My wife says that when I can’t work I get, “very testy.” Grant, however, outwardly was imperturbed. This is how Porter describes him this day:

General Grant Whittling Again – Civil War Reenactor Kenneth Serfass portraying Lt. General Ulysses S. Grant whittling at the site of Grant’s headquarters knoll at the Battle of the Wilderness. 150th Anniversary of the Battle of the Wilderness, 1864-2014

“… most of the day he sat upon the stump of a tree, or on the ground, with his back leaning against a tree. The thread gloves2)It is assumed that Mrs. Grant gave him the gloves. Grant wore them out whittling and never wore them again. remained on his hands, a lighted cigar was in his mouth almost constantly, and his penknife was kept in active use whittling sticks. He would pick up one small twig after another, and sometimes holding the small end away from him would rapidly shave it down to a point, at other times he would turn the point toward him and work on it as if sharpening a lead-pencil, then he would girdle it, cut it in two, throw it away, and begin on another.” – Campaigning with Grant

So, in summation, almost nothing has been done in the last six weeks to move development of the actual game (what we call the Game Engine application) forward. For this I can only sincerely apologize and repeat what you know: there’s only two of us coding it and when the lead coder falls ill everything stops.

What’s Going on with the Scenario Editor?

There are three stand alone programs that create data files used by the General Staff Game Engine: The Army Editor, the Map Editor and the Scenario Editor. The first two have long been available to early backers. What’s holding up the release of the Scenario Editor?

The Scenario Editor is actually done, and has been completed for some time. I use it to create the scenarios that you’ve seen and that I use to test the AI:

Screen shot of Antietam with battle groups, range of influence and objectives displayed. Click to enlarge.

So, why haven’t we released it? Because as we (that is to say, Andy) works on the Game Engine we discover that we need to make changes in the data files. For example, I left out the time it takes for a unit to change formations. To correct this, we need to add that value to the scenario data files and that, in turn, means a change to the Scenario Editor, itself. And, even worse, it means that previously created scenario files are no longer compatible and have to be redone. Andy, specifically, said he didn’t want to make the Scenario Editor available to early backers for this very reason: you could lose your earlier work.

So, in conclusion, again please allow me to apologize for these delays. There will inevitably be more delays before General Staff is completed. But, like Grant at the beginning of the Overland campaign, I am supremely confident in our inevitable success. Grant famously said at this time, “I propose to fight it out on this line if it takes all summer.”  It actually took all summer, fall, winter and next spring before Lee surrendered. But the ultimate success of the campaign was never in doubt.

And, as always, please feel free to email me directly at Ezra [at] RiverviewAI.com.

References

References
↑1 Ironically, the first person to use the Spanish word “derecho” to describe this type of storm was Gustavus D. Hinrichs, a German immigrant who settled in my hometown of Davenport in 1861.
↑2 It is assumed that Mrs. Grant gave him the gloves. Grant wore them out whittling and never wore them again.

Ty Bomba’s Primer on Strategy & Tactics

Leave a reply

Legendary wargame designer Ty Bomba.

I can think of no better introduction for Ty Bomba than his Wikipedia entry: “Ty Bomba is a prolific wargame designer from the United States. He is credited as the designer of over 125 board games or game items. At times between 1976 and 1988, Bomba held a security clearance as a certified Arabic and Russian linguist for the US Air Force, US Army, and the National Security Agency. In 1988, he was elected to the Charles Roberts Awards Hall of Fame. He was previously a senior editor at Strategy & Tactics Press. Bomba was co-founder and designer for XTR Corporation, a company that existed between 1989 and 2001. ” In other words, a very impressive career in wargame design and military strategy and tactical thinking.

Ty recently posted his Primer on Strategy & Tactics on Facebook and I asked his permission to repost it here, which he very kindly gave. I have spent much of my professional career trying to create computer algorithms for military tactics and strategy (a subject that I call ‘computational military reasoning’ and have written extensively about here). Ty has very succinctly stated much of what I’ve attempted to accomplish in his Primer below. Ty can be found on Facebook as ‘Ty Bomba’.

Ty Bomba’s Primer on Strategy and Tactics

Everything in strategy is very simple, but that does not mean everything is very easy” – Carl von Clausewitz.

Strategy Defined
A plan or policy intended to achieve a major or overall aim, and having to be
achieved in the face of opposition from others. All strategy is a contextual
interpretation of a problem and a compromised rationalization of a
solution. There are no formulas to end the tensions inescapably imposed by
uncertain intentions, faulty assumptions, unknown capabilities and vaguely
understood risks.

Laws of Strategy

  1. Know your own capabilities.
  2. Know your opponent’s capabilities and objectives.
  3. Pit your strengths against your opponent’s weaknesses.
  4. Prevent your opponent from pitting his strengths against your
    weaknesses.
  5. Never pit your strengths against your opponent’s strengths.
  6. Maintain an emergency reserve of five to 25 percent of your strength.
  7. Keep in mind your desired end-state: only do things that move you closer
    to it.
  8. Never repeat an already failed strategy with the expectation of getting a
    better result from it.
  9. The overarching objective of your strategy should be to create a state of
    surprise in your opponent. That uncertainty will delay, and otherwise make
    less efficient, his countermoves. That is a force multiplier for you.

Common Reasons for Strategic Failure

  1. Overconfidence due to previous successes.
  2. Analyzing information only after sifting it through the filter of dogma.
  3. Operating with insufficient reserves.
  4. Mirror imaging – using one’s own rationales to interpret the actions or
    intentions of an opponent – is the most common fault among decision
    makers.
  5. Objectives not well explained to those below the highest level of command.
  6. Objectives not adjusted according to new data coming from the
    operational environment.
  7. Unanticipated outside influences.

Tactics Defined
An action intended to achieve a specific end, undertaken while in contact with the
enemy.

Laws of Tactics

  1. Always seek to control the local high ground or its aerial or outer space
    equivalent.
  2. Move in short bounds from cover to cover so as not to be caught in the
    open by your opponent.
  3. Maneuver so as to engage your opponent on his flank or from behind and
    so as to prevent him from engaging you in that way.
  4. Don’t confuse “concealment” with “cover.” The former only gets you out of
    sight; the latter also offers protection from enemy fire.

Juncture of Tactics & Strategy
Your superior strategy can make up for your poor tactics; however, your superior
tactics will not make up for your poor strategy. As Sun Tzu put it: “Good strategy
combined with poor tactics is the slowest route to victory; good tactics combined
with poor strategy is just so much noise before your final defeat.”

Surprise
Surprise is a state of confusion in your opponent, induced by your introducing the
unexpected. At the strategic level, surprise is often viewed as the tool of the
weaker side, as the stronger side has the option of simply applying greater force.
At the tactical level, surprise is considered a force multiplier for the side causing it
by creating a temporary period of confusion and vulnerability in the surprised
force. Having multiple objectives lies at the heart of creating surprise in an
opponent.

The Most Difficult Thing
The most difficult thing in a dynamic situation is to know when to change
strategies. If you do it too soon or too often, you’re not a strategist; you’re an
opportunist. If you do it too late, or refuse to do it no matter what, again you’re
not a strategist; you’re a fanatic. Opportunists and fanatics are both easily
defeated by good strategists.

Movement & Maps

25 Replies

Screen shot showing unit movement arrows for the battle of Antietam scenario. Note how movement is stopped by the small creek in the upper left hand corner of the screen. Click to enlarge.

I‘m in self-quarantine. As many of you know, I had a bone marrow / stem cell transplant in 2014 followed by a year of intense chemo. I’m fine (all things considering) but my immune system took a beating. In 2014 H1N1 put me in the hospital for 3 days so I’m taking COVID-19 seriously as, uh, the plague. In a way, being cooped up in the house (the weather hasn’t been cooperating, either, as I had hoped to replace my daily gym workout with long walks with my dog, George) is a lot like a typical upper Midwestern winter: it’s what I call ‘good programming weather’. There’s not much else to do but hunker down and write code. So, obviously, I’m working on General Staff and I wanted to show you an update (above).

General Staff is being written in C# and Windows Programming Foundation (WPF). There are a number of technical reasons why this was a good idea but I’m not overly familiar with WPF and doing some basic things, like creating these transparent movement arrows, took far longer, and involved a lot more programming, than I thought it would.  My partner on this project, Andy O’Neill, is a WPF rockstar; I’m a novice. I would much rather be working on the ‘under the hood’ stuff (like AI) but for the last week or so I’ve been working on movement arrows while Andy is busy with another project (the kind that pays the rent). Anyway, I’m very pleased about how they turned out. Please feel free to write me with comments.

Movement arrows for units at Quatre Bras. Note how blue units are stopped by the stream feeding the large pond. Click to enlarge.

Forward movement of this blue division is stopped by this tiny creek.

But, I also discovered a very interesting problem while testing these movement routines: our maps are too good! If you look at this detail (right) from the Antietam screen capture (above), you’ll see that movement is stopped when it encounters a tiny creek. I’ve walked that area of the Antietam battlefield and that little creek (well, I think it would be more properly called a ‘crick’) wouldn’t stop a division moving forward. However, the movement validation routines stop units from crossing bodies of water (except in column formation while crossing a bridge or a ford).  Ed Kuhrt, who digitized these great maps and copied every small detail was, perhaps, too precise. Definitely better to err on the side of being too precise when it comes to maps, Ed.

We’re seeing the same thing at Quatre Bras (above): the little streams that feed Materne Pond (Etang Materne) are also stopping the French from attacking. I haven’t been to Quatre Bras but I know the French crossed the small streams to attack the Anglo-Allied army’s positions.

Luckily, fixing this is easy. If you have done any work with the General Staff Map Editor you know that erasing terrain features, like water, is quick.

Removing the water in the little stream that feeds Antietam Creek by placing a ‘field’ over the water. Note that the ‘Field’ object is ‘above’ the ‘River’ object on the left Edit Terrain tab. Screen shot. Click to enlarge.

In the above screen shot we’ve opened the Antietam map in the General Staff Map Editor and drawn a ‘Field’ over the ‘crick’ that kept the Union forces from advancing. Note that both the Field and the River are objects and whichever object is higher in the list on the left ‘overwrites’ lower objects. If, for whatever reason, we wanted to restore the water in the creek we could either delete the Field or move it lower down the list than the River object.

Edit: After originally posting this, some readers (see comments, below) suggested that units crossing a small stream should suffer a movement penalty. This is absolutely correct. Instead of ‘painting’ with ‘field’ terrain, I should have used ‘mud’. This allows for a movement penalty (set in the Scenario Editor).

Adjusting unit type speeds across various terrain types in the General Staff Scenario Editor. Screen capture.

And now that that obstacle to movement has been eliminated the I and XII Union Corps can advance:

Now that the water has been removed from the little stream feeding Antietam Creek the Union forces can advance again. Screen shot.

As always, I would love to hear any comments or feedback that you may have.

Introducing a New Generation to Wargames

26 Replies

My childhood friend, Carl Hoffman, introduced me to Avalon Hill wargames. Carl is now a history teacher.

It was my good friend, Carl Hoffman, who lived across the street when I was about 10, who introduced me to Avalon Hill (AH) wargames. The AH wargames of the 1960s were perfectly suited to spark a kid’s imagination. The rules were easy to understand (four pages, big type, with illustrations explaining movement and combat), the Combat Results Table (CRT) was straightforward (and taught us to calculate ratios, too), and we could refight Gettysburg or Waterloo on a rainy afternoon. We learned history (Carl became a history teacher) and problem solving (I became a computer scientist). I’m sure many of us had similar experiences forty or fifty years ago.

For a long time I’ve felt that there is a need for similar ‘introductory wargames’ to engage the next generation of grognards and wargamers. While the hardcore aficionados want more complex and detailed games I’ve also understood that we needed simple, introductory games, to entice a new generation.  From the beginning, I have always had a simpler wargame embedded inside of General Staff. Specifically, if we remove all the layers of historical simulation, what remains is a simple introductory wargame.

The Layers of Historical Simulation in General Staff

Each layer of historical simulation can be turned on or off when playing a General Staff scenario. The more options you add, the more historically accurate the simulation becomes. The options are:

  1. Unit strength
    1. Unit strength is a value from 1 – 4 with units being reduced in steps.
    2. Unit strength is the actual historic number of troops and every individual casualty is tracked.
  2. Combat resolution
    1. Simple Combat Resolution Table like the old AH CRT.
    2. Complex Combat Resolution Equation taking into effect morale, experience, leadership, terrain, and elevation.
  3. Moving units
    1. Units are moved directly by the player.
    2. Orders to move units are issued down a chain of command from the top HQ to the subordinate HQ via couriers and the rapidity with which the orders are executed depends on the Leadership Value of the subordinate HQ and subordinate units.
  4. Fog of War (FoW)
    1. No Fog of War. The entire map is visible and all units (friend and foe) are displayed on it.
    2. Partial FoW. The entire map is displayed and the sum of what all friendly units can see is displayed.
    3. Complete FoW. You see only what the commander can see from his HQ and nothing else. All unit positions not directly observable are updated via couriers and are frequently no longer accurate by the time the courier arrives.

So, at it’s most complex (let’s call this a Historical Accuracy level of 100%) this is what the player commanding the Army of the Potomac (Blue) would see (what General George McClellan could actually see through his telescope on the lawn of the Pry House on the morning of September 17, 1862):

Antietam from the perspective of General George B. McClellan at the Pry House on the east bank of the Antietam Creek. This is complete Fog of War and the highest level of historical accuracy. Screen shot. Click to enlarge.

And, interestingly, this view of what McClellan could see is confirmed in The U. S. Army War College Guide to the Battle of Antietam and The Maryland Campaign of 1862 edited by Jay Luvaas and Harold W. Nelson.”General McClellan and his headquarters staff observed the battle from the lawn of the Pry House… Through a telescope mounted on stakes he enjoyed a panorama view of the fighting… He could see Richardson’s division break through the Confederate position at the Bloody Lane. He could not, however, follow the movements of the First and the Twelfth Corps once they disappeared into the East Woods, which masked the fight for the Cornfield, nor did he witness the attempts to seize Burnside’s bridge to the south because  the view from the Pry House was blocked by trees and high ground.” – p. 119.

So, the above is the 100% historically accurate view of the battle of Antietam from McClellan’s Headquarters. This is the ‘introductory’ view:

Antietam in ‘Introductory’ mode. Note that unit strengths are represented by one to four icons. Also note the lack of HQs. Screen shot. Click to enlarge.

I first saw the concept of ‘unit steps’ in Jim Dunnigan’s Avalon Hill classic, 1914 and I’m shamelessly using it here. I very much like the simplicity of this system: as units take casualties, they are reduced, in steps, from four icons, to three, to two, etc. I also very much like the idea that this is not an abstraction of the battle of Antietam (or Little Bighorn, or Quate Bras, etc.) but the actual units in their actual locations. This fulfills my requirements for an introductory wargame: historic, teaches tactics and problem solving, easy to play, simple rules, quick to learn and quick to play (I would think a game could easily be played in less than an hour).

Here are some more General Staff scenarios in ‘introductory mode’:

1st Bull Run, 11:30 hours, ‘introductory’ mode. Screen shot. Click to enlarge.

Little Bighorn in ‘introductory’ mode. Screen capture. Click to enlarge.

Quatre Bras in ‘introductory’ mode. Screen capture. Click to enlarge.

I could use your help! Announcing a ‘name the mode’ contest!

From ‘The American Kriegsspiel. Clicking on this image will take you to the Grogheads.com article on William Livermore’s American Kriegsspiel.

I originally called ‘introductory’ mode, ‘Kriegsspiel’ mode because I was reminded of the maps and blocks that Kriegsspiel uses. However, I pretty quickly received some emails from the Kriegsspiel community complaining – and rightfully so – that Kriegsspiel isn’t an introductory game. Absolutely! And if you’ve ever taken a look at the original rule books and tables you would agree, too.

So, here’s my problem (and how YOU can help): I need a new phrase to replace ‘Kriegsspiel mode’. I’ve been using ‘Introductory mode’ but I just don’t like it. I really need a new name for this version. I’m open to any suggestions. How about a completely made up word? ‘Stratego’ would be great if it hadn’t already been used. So, I’m announcing a contest to ‘name this mode’. The winner will receive 2 General Staff coffee mugs. Please email me (Ezra@RiverviewAI.com) with you suggestions. Thank you for your help!

Feeding the Machine

10 Replies

The famous Turing Machine1)It was first described in Turing’s, “On Computing Machines with an Application to the Entscheidungsproblem,” in 1937 which can be downloaded here: https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf. Also a very good book on the subject is Charles Petzold’s, “The Annotated Turing: A Guided Tour through Alan Turing’s Historic Paper on Computability and the Turing Machine.” was a thought experiment and, until recently did not physically exist 2)Yes, somebody has built one and you can see what Turing described here: https://www.youtube.com/watch?v=E3keLeMwfHY . When computer scientists talk about machines we don’t mean the, “lumps of silicon that we use to heat our offices,” (thanks Mike Morton for this wonderful quote), but, rather, we mean the software programs that actually do the computing. When we talk about Machine Learning we don’t think that the physical hardware actually learns anything. This is because, as Alan Turing demonstrated in the above paper, the software functions as a virtual machine; albeit, much more efficiently than creating a contraption with pens, gears, rotors and an infinitely long paper strip.

When I talk about, “feeding the machine,” I mean giving the program (the AI for General Staff is called MATE: Machine Analysis of Tactical Environments and the initial research was funded by DARPA) more data to learn from. Yesterday, the subject at machine learning school was Quatre Bras.

Screen shot of the General Staff AI Editor after analysis of Quatre Bras and calculating the flanking Schwerpunkt or point of attack (blue square).  Click to enlarge.

The MATE tactical AI algorithms produce a plan of attack around a geographic point on the battlefield that has been calculated and tagged as the Schwerpunkt, or point where maximum effort is to be applied. In the above (Quatre Bras) scenario the point of attack is the extreme left flank of the Anglo-Allied (Red) army. I apply the ‘reasonableness test’ 3)Thank you Dennis Beranek for introducing me to the concept of ‘reasonableness test’. See https://www.general-staff.com/schwerpunkt/ for explanation and think, “Yes, this looks like a very reasonable plan of attack – a flanking maneuver on the opponent’s unanchored left flank – and, in fact, is a better plan than what Marhshal Ney actually executed.

It would be good at this point to step back and talk about the differences in ‘supervised’ and ‘unsupervised’ machine learning and how they work.

Supervised machine learning employs training methods. A classic example of supervised learning is the Netflix (or any other TV app’s) movie recommendations. You’re the trainer. Every time you pick a movie you train the system to your likes and dislikes. I don’t know if Netflix’s, or any of the others, use a weighting for how long (what percentage watched over total length of show) watched but that would be a good metric to add in, too. Anyway, that’s how those suggestions get flashed up on the screen: “Because you watched Das Boot you’ll love The Sound of Music!”  Well, yeah, they both got swastikas in them, so… 4)Part of the problem with Netflix’s system is that they hire out of work scriptwriters to tag each movie with a number of descriptive phrases. Correctly categorizing movies is more complex than this.

Supervised machine learning uses templates and reinforcement. The more the user picks this thing the more the user gets this thing. MATE is unsupervised machine learning. It doesn’t care how often a user does something, it cares about always making an optimal decision within an environment that it can compare to previously observed situations. Furthermore, MATE is a series of algorithms that I wrote and that I adjust after seeing how they react to new scenarios. For example, in the above Quatre Bras scenario, MATE originally suggested an attack on Red’s right-flank. This recommendation was probably influenced by the isolated Red infantry unit (1st Netherlands Brigade) in the Bois de Bossu woods.  After seeing this I added a series of hierarchical priorities with, “a flank attack in a woods (or swamp) is not as optimal as an attack on an exposed flank with clear terrain,” as a higher importance than pouncing on an isolated unit.  And so I, the designer, learn and MATE learns.

My main concern is that MATE must be able to ‘take care of itself’ out there, ‘in the wild’, and make optimal decisions when presented with previously unseen tactical situations. This is not writing an AI for a specific battle. This is a general purpose AI and it is much more difficult to write than a battle specific AI. One of the key aspects of the General Staff Wargaming System is that users can create new armies, maps and scenarios. MATE must make good decisions in unusual circumstances.

Previously, I have shown MATE’s analysis of 1st Bull Run and Antietam. Below is the battle of Little Bighorn in the General Staff AI Editor:

The battle of Little Bighorn in the General Staff AI Editor. Normally the MATE AI would decline to attack. However, when ordered to attack, this is MATE’s optimal plan. Click to enlarge.

I would like to expose MATE to at least thirty different tactical situations before releasing the General Staff Wargame. This is a slow process. Thanks to Glenn Frank Drover of Forbidden Games, Inc. for donating the superb Quatre Bras map. He also gave us maps for Ligny and Waterloo which will be the next two scenarios submitted to MATE. We still have a way to go to get up to thirty. If anybody is interested in helping to create more scenarios please contact me directly.

References

References
↑1 It was first described in Turing’s, “On Computing Machines with an Application to the Entscheidungsproblem,” in 1937 which can be downloaded here: https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf. Also a very good book on the subject is Charles Petzold’s, “The Annotated Turing: A Guided Tour through Alan Turing’s Historic Paper on Computability and the Turing Machine.”
↑2 Yes, somebody has built one and you can see what Turing described here: https://www.youtube.com/watch?v=E3keLeMwfHY
↑3 Thank you Dennis Beranek for introducing me to the concept of ‘reasonableness test’. See https://www.general-staff.com/schwerpunkt/ for explanation
↑4 Part of the problem with Netflix’s system is that they hire out of work scriptwriters to tag each movie with a number of descriptive phrases. Correctly categorizing movies is more complex than this.