Yagmur Denizhan: Performance-based control of learning agents and self-fulfilling reductionism.

Yagmur Denizhan: Performance-based control of learning agents and self-fulfilling reductionism. Systema 2 no. 2 (2014) 61-70. ISSN 2305-6991. The article licensed under the Attribution-NonCommercial-NoDerivatives 4.0 International License. A PDF file is here.

Abstract: This paper presents a systemic analysis made in an attempt to explain why half a century after the prime years of cybernetics students started behaving as the reductionist cybernetic model of the mind would predict. It reveals that self-adaptation of human agents can constitute a longer-term feedback effect that vitiates the efficiency and operability of the performance-based control approach.

From the Introduction:

What led me to the line of thought underlying this article  was a strange situation I encountered sometime in 2007 or 2008. It was a new attitude in my sophomore class that I never observed before during my (by then) 18 years’ career. During the lectures whenever I asked some conceptual question in order to check the state of comprehension of the class, many students were returning rather incomprehensible bulks of concepts, not even in the form of a proper sentence; a behaviour one could expect from an inattentive school child who is all of a sudden asked to summarise what the teacher was talking about, but with the important difference that –as I could clearly see– my students were listening to me and I was not even forcing them to answer. After observing several examples of such responses I deciphered the underlying algorithm. Instead of trying to understand the meaning of my question, searching for a proper answer within their newly acquired body of knowledge and then expressing the outcome in a grammatically correct sentence, they were identifying some concepts in my question as keywords, scanning my sentences within the last few minutes for other concepts with high statistical correlation with these keywords, and then throwing the outcome back at me in a rather unordered form: a rather poorly packaged piece of Artificial Intelligence.
It was a strange experience to witness my students as the embodied proof of the hypothesis of cognitive reductionism that “thinking is a form of computation”. Stranger, though, was the question why all of a sudden half a century after the prime years of cybernetic reductionism we were seemingly having its central thesis1 actualised.


1 thought on “Yagmur Denizhan: Performance-based control of learning agents and self-fulfilling reductionism.

  1. My comments are on a few themes which appear within the paper. They are stand-alone, selected on the basis of curiosity, and do not necessarily present a coherent over-arching argument.

    On Games:

    “…the winning strategies in such games were typically based on identifying the underlying algorithm instead of being “misled” by the story.”

    This is very true, and I view it as a result of the human tendency to simplify or reduce puzzles to their essence.

    The most notable example is what is referred to as “grind” in Massively Multiplayer Online Games (MMOGs). This refers to the player experience that follows the efficient mastering of repetitive tasks which produce steady, predictable basic income or rewards. The winning strategy is often an absurd mockery of the scripted challenge. The condensed objective is profitably to exploit or defeat the game mechanics. Game creators are constantly forced to play the role of a financial regulator, “rebalancing” the game via parameter or mechanics changes, as players adapt. This is even exploited by organised criminals who “farm” items of value and sell them online, converting them to real-world currency. They are not misled by the story. They take shortcuts through it.

    A stark example from the Real Time Strategy (RTS) gaming world is Battlezone 2, which offered immersive command of squadrons of hover tanks, bases, supply logistics, and so on. Unfortunately, because the tank AI cannot see a lone infantryman except in a very narrow field of view, past a certain point almost every level could be defeated by jumping out of one’s tank, hiking up a mountain and calling in air strikes. The majority of the game’s features were thus rendered redundant except in a few highly contrived scenarios.

    I am willing to bet that monkeys, perhaps even rats, can be shown to display similar behaviour. Give them a puzzle, teach them a sequence of actions which release a food pellet, and they should swiftly learn to omit those steps which are actually redundant. Indeed, I would be surprised if there are not already published studies along these lines.

    Already at the level of secondary school exams, many of my contemporaries discovered that playing to the exam marking scheme, rather frustratingly, turned out to be the winning strategy. By requesting the return of exam scripts, some students were disappointed to learn that non-specialists awarded marks by keyword-matching exam scripts to the marking scheme.

    On the Capital Markets

    Following the 2006-2008 financial crisis and the advent of political intervention* and near-universal adoption of algorithmic trading, the capital markets have become just such a game. Trading based on economic fundamentals and a close reading of financial statements and regulatory filings has become foolish, even irresponsible. Not only has trading become the pinnacle of the application of system control theory, but “identifying the underlying algorithm” of competing agents and tricking or exploiting them for arbitrage, has become a legitimate strategy. The wider economic context or story is rendered largely irrelevant except as a known input state to the target algorithm.

    * By this I mean the relatively recent (in the US and EU) extension of political interest to, and intervention in, intra-day price movements. Algorithmic traders scan newswires in much the same manner as the description of the professor’s students, and operate pre-defined protective mechanisms accordingly. This is markedly different to historically accepted forms of interest by political and regulatory authorities in macro-economics.

    On the Classification of Meaning

    “…given the target to reduce the number of patients waiting on trolleys – simply took the wheels off the trolleys, and reclassified them as beds…”

    I believe that bizarre examples of management action such as this are a symptom of the widespread adoption of Enterprise Resource Planning (ERP) software, by large organisations. The key to successful implementation of one-size-fits-all software such as this across a broad range of organisations, is not to customise the software, but to change administrative processes and workflows in accordance with it. This is what expensive consultants actually admit! Thus, at the outset, the finite and rather limiting set of criteria available within the software, come to redefine reality and the user’s freedom of action in response to it. A hospital manager finding himself or herself working with the same tool as an online retailer’s warehouse, will have seconds in which to make a decision, which is constrained by the software and its associated databases, yet freed from human supervision. The result might not be socially useful, but drop-down menus relieve the user from what might otherwise be a difficult and time-consuming set of decisions. What is more, he or she is very unlikely to possess the authority to circumvent the software. For this reason, the catchphrase “computer says no” has entered popular culture. Note that this is a relatively recent addition to management methodology. Prior to the 1990s, professional experience, intuition, unspoken convention, and deference to human management authority would have guided such actions.

    On System Models and Control Loops

    “It is worth noting that, given a fixed error correction strategy, the success of a closed-loop controller still depends on the correctness of the system model within it.”

    “…another factor needs to be taken into account: the system has its own internal management.”

    I suspect that when we are dealing with humans, it is the error correction strategy that is more amenable to change, by external influence (that is, by intervention from outside the feedback loop), than is the self, world or system model, but I will have to think some more about it.

    “It is also worth noting that the internal management system operates to a great extent at subconscious level.”

    Perhaps when it operates at a conscious level, it is characterised as ruthlessness or deviousness? What would an individual with flexible world and self models, who has successfully modelled the external management system, resemble? To me it brings to mind Jeremy Irons’ character in Margin Call, the insolvent investment banker who consciously redefines success as lesser failure relative to soon-to-be-sabotaged others. It appears to me that he wins the adaptation game and the control game. The consequences for the wider system are negative.

    “…reached a critical level of agreement such that I could not help noticing the resulting constructive interference.”

    Destructive interference is an equally remarkable phenomenon: when the world and/or self models imprinted on the individual are incompatible with the system model imposed by management (that is, by the external world). A divergence of values and lack of shared concepts results in ungovernable / unteachable human agents.

    “Thanks to this alleged objectivity the external performance assessments can push the immediate internal observation of the own state aside, and can enter a dangerous interaction with the subconscious levels of the internal evaluation system, trigger feelings of shame, guilt and inferiority and allow the external evaluation criteria to be internalised. The danger is even larger if –as frequently done– superiors’ performances are indexed to those of their subordinates such that the hierarchical pyramid is both reinforced and used against each member at every level of the hierarchy. Consequently, systematic exposure to such a management scheme can leave permanent traces particularly on the self-model of the human agent, reducing it to a much simpler one that can be described in terms of the performance criteria defined by the management system.”

    This description elegantly matches the working culture of many large corporations.

    “…a symptom I call “score addiction”.”

    An equally valid outcome could be called “score rejection”. Those who have experienced a management culture such as described in the preceding text, can reject external performance feedback as noise.

    “Once the mentality that considers performance criteria as ends-in-themselves reaches a critical level of prevalence…”

    Remuneration influences this too. Anecdotally, few would argue that remuneration functions as an input parameter (gain on an amplifier?) that influences the quality or extent of control exercised by the human agent. This is the “I don’t get paid enough to make the extra effort” rationalisation beloved of junior functionaries across the world. But many studies have shown that past a threshold, financial reward has its limits as an effective incentive.

    What to Change?

    I would say that “system collapse” as the author puts it, manifests itself as organisations, be they corporate or governmental, falling into insolvency. The threats posed by stifled creativity or dysfunctional governance are well known and widely written about, but perhaps only superficially understood, because the failure of institutions is a constant feature of life.

    Interestingly, by default, ERP systems assume 85% time allocation of people to task over a working week. The rest could be called “friction”: general administrative work that is not specifically productive. “Creativity seminars” and the like, attempt to put to productive use time which does not exist, or is otherwise not available. Yet some technology companies set aside one working day per week for employees to pursue speculative side-projects, and it speaks volumes about management culture that this space for creativity is considered to be radical, or the exclusive domain of start-up culture.


    Lastly, consider that this generation has grown up in a cynical age, in which the highest-paid private sector activity, and most influential public policy, not to mention entertainment, unashamedly consists of fooling machine intelligence. In surveying this world, what impressions might students be expected to form?


    That concludes my thoughts. I liked the article, it was thought-provoking and there was much that I recognised.

    “The mechanisation of the human” is treated as unfortunate when it manifests itself, but as a conceptual framework, it can be useful, when people can be understood in no other way.

    Does the author’s university have a business school? Perhaps a modified version of this paper might be accepted for publication in one of their periodicals.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s