<section data-background="akram-shehadi-425084-unsplash.jpg">
<table style="width:100%">
<tr>
<td align="left">
<h2 style="color:#ffffff">what the machine saw:</h3><br><h3>Studying the trade in human remains<br> in an era of big data</h4></td>
</tr>
</table>
<p align="right" style="color:#ffffff"> <small> Shawn Graham<br>Carleton University <br> @electricarchaeo <br> follow along at bit.ly/sg-york
</p>
---
<section data-background="Ottawa_Valley.jpg">
Note:
Would begin with territorial acknowledgement, if I was in Canada. But the material I study is traded through the internet; the trade transgresses borders, it transgresses time. So I will acknowledge the land on which Carleton stands. Because those people: those people have had their ancestors bought and sold. Spoke with a scholar in sweden last month, lovely fellow, who just didn't see why anyone should care about bones: they do not mean anything to him, therefore it follows that anyone feeling different is just misguided. They're just calcium in interesting arrangement.
But they're not.
---
Abraham Ulrikab
Note:
Abraham's story
---
How many more Abrahams are out there?
Note:
And so I acknowledge the land to acknowledge the people, to acknowledge that my academic ancestors and perhaps my literal ancestors bought and sold human remains in the pursuit of a dubious goal: the construction of whiteness as the pinacle of human evolution.
---
## tl;dr
A _lot_.
Note:
"By taking skeletons out on backs we got them out without Indians realizing the bulk & so free from objections. But when the Indians return from fishing it would not be pleasant to be there.""
- Smith to Boas, 19 August 1899, AMNH
---
## The Plan for Today's Talk
- what we want to do: teach machines to see like archaeologists
- why we want to do it: the bonetrade project
- begin with text and context
- experiments with vision
- ethical issues of this entire endeavour
Note:
This talk is about me trying to be ethical. how I come to study this stuff. what we've already discovered. what moral and ethical questions the work has already raised about doing archaeology, and how archaeological codes of ethics need to adapt
---
I will sometimes be showing images of human remains.
Many archaeological codes of ethics frown upon this.
---
## Canadian Archaeological Association
<p align="left">"The CAA requires authors to obtain documented permission from descendant community(s) to present human remains in any media or form.<br><br>
Beyond genetics, descendant communities are defined here by their historical, cultural, and symbolic associations to places they consider ancestral."
</p>
[source](https://canadianarchaeology.com/caa/about/ethics/presentation-human-remains-caa-media)
---
And so that brings us to our first ethical dilemma: <Br><br>our codes of ethics don't imagine a world where human remains _and images of human remains_ are traded like so many pokémon.
Note:
In many ways, our current digital moment is a rerunning of 19th century 'science'
---
Note:
when people want to buy bones, where do they go?
when people want to sell bones, where do they go?
when people like or repost images of the dead.... why?
where and from whom did these remains originate?
...and can machine vision or big data approaches help answer these questions?
Note:
given that much of this trade is morally dubious and legally in various grey zones
---
<section data-background="/papers/stockholm-04-25/joseph-chan-264837-unsplash.jpg">
<table style="width:100%">
<tr>
<td align="right">
<h2 style="color:#ffffff">teaching machines to see like archaeologists</h2><Br><h2>presumes we know how archaeologists _see_</h2></td>
</tr>
</table>
Note:
which presumes that we know how _archaeologists_ see
---
## narrator:
## we don't
---
<section data-background="/papers/stockholm-04-25/abigail-low-532799-unsplash.jpg">
Note:
perry, hamilakis on vision, seeing, reification
roopika risam on the kinds of humans our visualizations etc permit
visualizations in the service of scientific racism
grounded & ungrounded antiquities
try to talk about connaisseurship
pompeian wall styles
human remains
---
## What it would be _nice_ to be able to do
1. teach the machines to understand _style_ & _composition_
2. teach the machines to spot the differences between a forgery & something authentic
3. send the machines out there to do the spotting at a scale and speed that we could never hope to achieve with our own fleshy fingers & imperfect eyes
---
<img src="readingmachines.png" height = 75%>
Note:
And so we set out to study the trade in human remains.
We began by looking at the _text_ of posts because, well, <br> we didn't know how to look at images _like a machine does_.
But maybe we could _read_ like a machine.
---
> <small>'I have a pile of teeny human skull scraps laying around. Due to etsys rules i cannot sell human bone or make a listing but id love to do custom orders for anyone interested in a pendant, ring, etc made from a human skull fragment. Dm me! #bone #bones #skull #humanbone #humanskull #fragment #skullfragment #oddities #oddity.'
https://www.instagram.com/p/znHyR7AbXS/
<small> see the excellent thread by [Linzi Harvey](https://twitter.com/HarveyArchaeo/status/1093836557335912448) on the legalities of buying, selling human remains in the UK, btw </small>
Note:
Not everyone is as bold as this. Most are aware that their _posts_ can be searched (ie key-word searching), and so rely on more subtle cues
On Instagram alone _where dollar values were named_
+ total amounted to approximately $190,000
+ the value of private sales cannot of course be known
+ Cultural impact: no.s of followers, following of _just those accounts that name a price_
+ a network with 138,014 individuals connected by 172,208 links
association with taxidermy, tattoo parlours - do i still have that cardiff photo?
three broad groupings: specialists explicitly interested in bones; generalists who follow the bones but also other things; and people who don't post pictures themselves but follow the bone accounts
---
Note:
video of man in Belgium degreasing a skull. Likely source for such a skull? WW1 battlefield. He's preparing it for cutting into an exploded view.
desecration of ww2 battleship wrecks - steel is pre 1945, so different atomic signature thus valuable; bones collected, given to the boss... what happens to them?
How do we deal with it?
Part of the problem is that not all of this is _illegal_. approaches and issues are of a part with broader trade in antiquities. Also: human remains are not property under common law!
broader trade tends to focus on the auction houses and the big collectors; smaller objects, ebay, etc do not attract as much attention
but the techniques developed to deal with these smaller items could be of significance for studying the broader trade, and the penetration of archaeological consciousness into the broader publics
(see excellent thread by [HarveyArchaeo](https://twitter.com/HarveyArchaeo/status/1093836557335912448) on the legalities of buying, selling human remains in the UK)
---
## digital archaeology
### meets
## digital humanities
Note:
digital humanities approaches to data mining try to understand the contextual significance of what has been extracted
in our work, we have in the first instance scraped ~ 13 000 posts from one calendar year on Instagram (one of many places where human remains are traded); some posts were caught in the trawl as far back as 2013
we focussed on the _language_ the posts, building topic models and word vectors to understand some of the rhetorical constructions of buying and selling human Remains
---
Note:
Exploring the vector space defined by binary pairs, 'good'-'bad', 'for sale'-'not for sale’. Hints of various dynamics at play can be viewed in contrasting, but related, terms, e.g. as with ‘acrylic painting’ falling in negative space and ‘skull decor’ falling in positive space. This might be a glimpse into the taste or aesthetic of those who create skull-related art, which may be important for understanding what creates the taste and demand for human remains themselves.
literal posts, 'human skulls for sale' do exist; but often the language is coded, more oblique, and seems to depend on the composition of the image itself to signal something for sale
---
---
<table style="width:100%">
<tr>
<td align="left">
</td>
<td align="right">
</td>
</tr>
</table>
Note:
the close association of topics clearly connected with buying and selling, and topics related to trophy skulls and other indigenous ethnographic materials. While a post might not necessarily mention a trophy skull being bought and sold, the patterns of discourse are very similar. These materials are not being discussed with respect, but as commodities. As we progress to the right in the dendrogram, we see other topics connected with the mechanics of the trade - giveaways, a gift for christmas, hallowee'en.
---
Nothing says romance like giving your beloved a skull.
---
<section data-background="post-metadata.png">
Note:
Lots and lots of data. So we're still collecting that kind of post metadata. But it's not as simple or straightforward as scraping the full metadata.
---
## The Parable of the [Viral Hippo](https://www.buzzfeednews.com/article/alexkantrowitz/people-are-turning-their-accounts-into-bots-on-instagram)
Note:
A photo of a black square gained thousands of likes from real people, verified accounts, within hours of being posted. Result of using a service called Fuelgram. "for a few dollars a month and access to your Instagram log-in credentials, it will use the accounts of everyone who paid that sum to like and comment on your posts — and it will use yours to do the same to theirs.
In other words, Fuelgram creates fake engagement from real Instagram accounts. And it's quite effective. Fuelgram makes posts appear more popular than they are, tricking Instagram's algorithm into spreading them further, sometimes right into the service's high-profile Explore tab. And there's a reasonable chance there's one in your feed right now, because Fuelgram is just one of a number of Instagram-juicing services available today, and the photo-sharing platform's engagement-rewarding algorithm incentivizes people to use it."
---
<section data-background="federico-beccari-62960-unsplash.jpg">
Note:
Which means that studying the text of posts themselves - counting likes and so on - might not be as straightforward as we thought.
But maybe the images themselves - the way they're composed, their subject matter, influences and similarities - maybe that's the way forward.
---
## taphonomic data processess
Data aren't raw, they're cooked
Data aren't things given, they're things created
And this applies to the tools, too.
---
### how neural networks beget computer vision
<img src="https://media.giphy.com/media/LXRhyTjXJEUmQG1NrT/giphy.gif" height = 400 px>
[Alex Mordvintsev](https://twitter.com/zzznah/status/1118917356430995456)
---
Note:
So we ran off madly in all directions, downloading tutorials, slapping our data into various frameworks....
---
## Labelling Images: Can we find human remains automagically?
<img src="/papers/fields-02-19/fig1-huffer-wood-graham-conch.png" height="200"/>
<img src="/papers/fields-02-19/fig2-huffer-wood-graham-mask.png" height="200"/>
Note:
Not with the default neural network models that Google gives us. Also note that these images show us just what it is the neural network is attending to.
---
---
---
### Training Our Own Classifier
[Build Your Own Classifier Tutorial](https://bonetrade.github.io/tutorials/tensorflow-for-poets/)
Note:
It's *super easy* to do this: the serious how-to research is racing ahead *so fast*. I built that classifier in an afternoon. 5 years ago? this would've been the stuff of dreams. Now imagine such a tool in the hands of a dealer.
So we slowed down a bit. Started experimenting with various technologies not to see what they could do, but to see how they _broke_
---
<section data-background="oskar-yildiz-631351-unsplash.jpg">
# Experimental Digital Archaeology
---
## experiment #1
### eye spy the fonseca bust
---
One-shot learning
- pass an image through a neural network
- pass a different image through an identical network
- measure the difference between the results. The difference is the degree of similarity
- requires much less data
- requires careful training data
Note:
"In face recognition systems, we want to be able to recognize a person’s identity by just feeding one picture of that person’s face to the system. And, in case, it fails to recognize the picture, it means that this person’s image is not stored in the system’s database.
To solve this problem, we cannot use only a convolutional neural network for two reasons: 1) CNN doesn’t work on a small training set. 2) It is not convenient to retrain the model every time we add a picture of a new person to the system. However, we can use Siamese neural network for face recognition." https://towardsdatascience.com/one-shot-learning-face-recognition-using-siamese-neural-network-a13dcf739e
---
Training on folders of images of Roman sculpture
Pairs left to right, 1 = different, 0 = same
---
Q. Is the Fonseca Bust a Flavian Woman?
Pics of Fonseca in one folder, pics of a flavian woman in the other
---
<img src="/papers/stockholm-04-25/fonseca-flav.png" height="300"/>
Looks pretty good! But wait...
---
<img src="/papers/stockholm-04-25/fonseca2.png" height="400"/>
---
## experiment #2
### give me some architectural ornamentation, and make it in pompeian red
---
generative adversarial networks
<small>Dev Nag, https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f </small>
Note:
The idea is to learn what is important in an image by getting the forger to successfully create new images; the things that it makes show what the detective considered to be the signature elements.
---
etc
---
<img src="/papers/stockholm-04-25/gan-pompeii.png" height="300"/>
<img src="/papers/stockholm-04-25/gan-pompeii2.png" height="300"/>
This _looks_ good, but the same visions persist, and it's actually just learned to reproduce the training set. It is not _composing_ new images. Training set was too small.
---
So I fed it a bigger sample from our bone trade research:
---
## experiment #3
### is this a skull I see before me?
---
<small> 10k images tagged with 'skulls for sale' and similar, collected Feb 2019, visualized with Pix-Plot, Yale DH lab, [code repo](https://github.com/YaleDHLab/pix-plot) </small>
---
---
---
<img src="/papers/stockholm-04-25/bone-lust-eigenvector-illustration.png" height = "400" />
Note:
Time stamps. Is there directionality in the influence?
left vertex of triangle: Fri Feb 26 18:57:31
next in tail: Tue Mar 11 23:19:25
next in tail: Sat May 10 20:09:42
last in tail: Sat Apr 12 21:40:37
This seems to work so we're scaling it up. Stay tuned.
---
# So What?
Note:
teaching the machines to see works in a way that suggests greater authority than is warranted, which is dangerous.
---
<small>Bailey, J. 2017 'Machine learning for art valuation. An interview with Ahmed Hosny', Artnome.com, https://www.artnome.com/news/2017/12/2/machine-learning-for-art-valuation </small>
Note:
Already, people are using neural networks to identify 'high value' artworks so as to set reserve prices
---
<small>Hirschman, Dan. artificial intelligence discovers gayface. sigh. https://scatter.wordpress.com/2017/09/10/guest-post-artificial-intelligence-discovers-gayface-sigh/</small>
Note:
Already, people are using neural networks to identify sexual orientation, or presence/absence in criminal databases
---
<section data-background="/papers/stockholm-04-25/josh-marshall-133093-unsplash.jpg">
Note:
Already, the use of these techniques reifies assumptions about culture, origin, and value that dehumanize and demean the living _and the dead_.
---
<section data-background="/papers/stockholm-04-25/kyle-glenn-336141-unsplash.jpg">
Note:
The use of this technology in the context of antiquities trading, human remains trading, forgeries is going to throw up many _edge cases_ that will require human intervention to sort out. This may improve the algorithm: but will it make it ethical?
An ethical use case: taxonomic processes. Instead of authenticating, let the machines look for hints of context. Let the machines look for low-stakes features.
---
## ethical dangers of machine vision
1. reification of a privileged point of view of what constitutes 'style' or 'material culture'
2. automated authentication to raise the price of antiquities and fuel the trade
3. replication of the sins of scientific racism
---
<h2 align="right"> this whole project <br>is rife with <br>ethical <br>dilemmas </h2>
<section data-background="/papers/fields-02-19/markus-spiske-1165564-unsplash.jpg">
Note:
summary of the problems we see
-The reasonable right to privacy even if social media materials are posted openly online.
-The potential to identify individuals from their patterns of posting.
-The potential for CNN to be responding to extraneous other signals in the data rather than the ostensible subject matter.
-The potential for our own unexamined biases to be ‘baked-in’ to the training data we select.
-The potential for any classifier we make to be used to ‘authenticate’ or otherwise heighten the commercial value of the materials depicted.
-The potential for the technology to replicate the sins of the past when it comes to the ethical and just treatment of the dead.
-The potential for the technology to be harnessed by law enforcement in ways that work against social justice.
---
<section data-background="/papers/stockholm-04-25/callum-shaw-557874-unsplash.jpg">
Note:
Leave the complex reasoning to the humans.
Neural networks are too easily spoofed while also carrying too much rhetorical power.
for previous slide
Our funders are interested in us being able to identify descendent communities for the purposes of repatriation. Can we really do this? And if we are able, our experience in building classifiers so far just points out that what we believe about the images is easy to encode: if we say these are 'mowhawk', they become reified as mohawk. Finding visual tropes is one thing; saying that, beneath the tropes, this skull was a member of group x, y, z is I think a dangerous line to pursue. But is the good of repatriation worth the risk?
---
<section data-background="/papers/stockholm-04-25/callum-shaw-557874-unsplash.jpg">
<h3 align="left">thank you</h3>
<p align="left">shawn.graham@carleton.ca</p>
<p align="left"><small>This research has been generously supported by the<br> Social Sciences and Humanities Research Council of Canada</small>
<h3 align="left">[bonetrade.github.io](http://bonetrade.github.io) </h3>
---
### Creators on Unsplash.com
Akram Shehadi|Joseph Chan|Abigail Low
Federico Beccari|Oskar Yildiz|Josh Marshall
Kyle Glenn|Callum Shaw