promethea.incorporated

brave and steely-eyed and morally pure and a bit terrifying… /testimonials /evil /leet .ask? .ask_long?

sinesalvatorem:

blastfarmer:

sinesalvatorem:

cinefeminism:

quasitree:

sinesalvatorem:

From the latest SCC links post:

Freddie deBoer writes a white paper supporting standardized testing in colleges. His position is that private colleges need to be held accountable and we need proof that online courses don’t work, but American Interest points out that it might break the power of education-industrial complex if people who go to less prestigious institutions have an objective way to prove they’re just as good as people who went to more prestigious ones. And I will add that it might incentivize colleges to admit based on something vaguely resembling merit if they want higher test scores. Overall this would be amazing it it happened.

Um, but… We already have this in the Caribbean? Because we’re halfway civilised? Do you mean to tell me Americans are barbarians who let random colleges do whatever? If you don’t test your colleges, what are they even for? How can you know how a student at one college compares to one at another? WHAT IS THIS MADNESS???

I think I’ve determined the main trade-off between the Caribbean and the US: Sure, we’re poor; but when we design an institution, it isn’t fucking stupid.

I know I’m biased as someone who is, I think, reasonably smart and capable and does v poorly on standardized tests, but the idea that more standardized tests will force institutions to admit based on merit is ridiculous ime. Like with subject GREs, it will give schools an easy but inaccurate metric so they can avoid actually judging merit or potential.

As someone who does well on standardized tests, I also think that they are pointless as actual assessments. IMO the things that may make me successful in academics are exactly the things I had to learn to suppress to pass standardized exams. Logic, analytical skills, creativity. I think @quasitree is just not as good at suppressing her brilliance / perhaps her brilliance surpasses mine (the latter has always been my theory).

…Are your standardised tests really really different to mine? Like, as far as I can tell, logic, analytical skills, creativity, and depth of knowledge of the subject area are what I’ve been tested on?

Like, typical chem question would be something like: “You run a laboratory and have been given samples X, Y, and Z to identify. Their characteristics are [description]. Your budget is limited, so you can only devise one testing regimen to apply to all the samples. You can use some or all of [list of apparatus]. Design and justify an identification process with explanations of the principles behind each stage of chemical testing, cost effectiveness, the trade off between accuracy and efficiency, and the level of confidence you can expect for your identification.”

[cue me furiously writing four pages while cursing several deities]

I mean, if building and justifying your own experiments from scratch isn’t evidence of understanding chemistry, is their literally any observation you could make that would give you information about someone’s chemistry proficiency? If not, I still have to wonder what colleges are for because, for all we know, they aren’t teaching shit and no one can tell.

This response is based on one year of high-school chem with an above-average teacher in a 30-person classroom, and one semester of college-level intro chem in a 350-seat lecture hall (twice-weekly lectures, with a corresponding twice-weekly 30-seat ‘recitation’ period to explain the lecture and a once-weekly 30-seat lab period). This is also drawing on years of PSAT and SAT tests, and on the AP Calculus AB test.

In my experience, in the US, standardized tests are designed to make the jobs easy for the graders.

Your “typical chem question” would be the end-of-test essay question, and would be preceded by 25 to 50 multiple-choice questions requiring you to regurgitate facts about various compounds and their properties. Then, there would be around 25 multiple-choice questions where you would solve chemistry-related math or balance chemical equations. These two sections would all answered by filling in the bubbles on scantrons. If there is a short-form free response series of questions, (if!) then there will probably be no more than 5 of the free-form response questions, and your answer must fit within the designated area on the form. Your final answer (if it is a math problem) must have a box drawn around it: This is a standardized test. Priority is placed on the graders’ ability to grade your test.

Finally, there is the longer essay, which is again limited by space. When I took the SAT, the writing space available for the essay question on the writing section was about equal to one and a half single-sided pages of college-ruled paper. It was not a lot of space to write a 1-3-1 essay, and if your response wasn’t in the 1-3-1 format, if you were taking space to explain your arguments and to back them up with knowledge not included in the test form (this practice is encouraged), then you are more likely to run out of paper. If you write large, you’re more likely to run out of paper.

In the AP Calc AB exam, there was still a lot of bubble-filling. But there were more free-form responses, and at the end were six word problems, of which you were allowed to choose three. Each response was allowed to take up half a page of the printed workbook. You could use scrap paper to work, but all logic required for your response needed to be in the workbook.

The chem course I took in college was the not-remedial intro-to-chem course. You were expected to have memorized the speed of light, and you were expected to use 300,000,000 m/s as an acceptable substitute, because it would simplify your calculations and allow you to finish the test faster.

About half the class did not finish the average exam. The average exam grade was 75% correct, a ‘C’. ‘C’ is a grade in the middle of the scale from the top scores, marked ‘A’, to any failing grade, marked ‘F’. ('E’ is skipped for legacy reasons, except where 'E’ is used in place of 'F’ because 'F’ hurts children’s feelings. I’m not kidding.) (Letter grades are still tabulated from point scores in order to convert point scores to GPA scores, which are a 0-4 scale that is used in college admissions, in inter-school comparisons, and in entry-level job placement.

That chem course was explicitly designed as a weed-out course, to deflect people from the Chemistry department’s majors.

In comparison, my high school: with ~65 people taking the course in three sections, the (delightfully sadistic) teacher didn’t use scantrons. (It would not be cost-effective at this scale.) There might be 10 multiple-choice or true-false questions at the start of the text or exam, but the remainder of the test was all short-answer or essay questions, where you were expected to explain your reasoning. If there was an extra credit question, you were expected to be creative in your response.*

This teacher’s methods would not work in standardized testing, because this teacher’s methods required effort on the part of the grader. The grader would need to understand the question, the solution, the student’s answer and the student’s reasoning behind their answer.

Your “typical chem question” can’t be scaled up to deal with ~10 million students per year.

And thus, why standardized tests suck.

* One Intro to Engineering final exam was one question, whose prompt was as follows**: You awaken to find yourself in a locked room the size of this classroom (25 feet by 75 feet). There is a door in one corner. At the opposite end of the room is a weighty metal desk. On the desk is a cardboard file box containing one (1) fully-disassembled AK-47 and one round of ammunition. Next to the box is the AK-47 user manual, printed in Swahili. In 15 minutes, a hungry Bengal tiger will be introduced to the room through the door. Explain how you will survive the tiger and escape the room.

** If you recognize this teacher, go say hi! He likes hearing stories of our adventures.

For college-level standardised tests in the Caribbean, there would be two separate papers per subject area, plus a portfolio.

For the first paper, which receives lower weight in your grade, you would have to answer 60 multiple choice questions. If this is chem, they’ll be split between chem math and “what would you expect to happen if [chem thing] were done?” type of stuff. This is generally ~25% of your grade. It’s not thought too highly of and mostly exists as a check on the other bits. If you do really well on one type of assessment while flunking the others, someone calls your school and asks what’s going on.

Paper II would have 2-3 questions of the type I described, plus 20-30 (depending on subject) short-answer questions that would definitely require creativity to figure out. This would be ~35% of your grade.

Finally, there’s the School Based Assessment. This is a portfolio of projects that the Caribbean Examinations Council required you to do as part of your class during the year. In the hard sciences, this is mostly a write up of labs (some of which you design). In the social sciences, a research paper. In Computer Science, code. This was the final ~40%.

Then, at the end of the year, your school ships all this stuff to CXC head quarters. Additionally, since this is the summer vacation, some of your professors will also get shipped to CXC head quarters. There begins the super labour-intensive multi-week process of grading every exam paper and portfolio.

The papers are divided up and distributed among the graders. Physics professors get the physics papers; lit profs get lit; etc. Everything is broken up into pieces and assigned a serial code so you don’t know whose paper you’re grading.

If you are confused by an answer or find it ambiguous or think it doesn’t fit the rubric, you flag it as such and a panel of other professors in your field will look it over and come to a consensus grade. A couple of the grades you give will also be randomly submitted for review to make sure you aren’t grading too far out of line with the rubric.

After all this is done, the grades are distributed to the colleges. Then the colleges hand grade slips to the students. Then the professors are allowed to go home. I have been told that, if the review panels think you’ve been marking unfairly, they drown you in the caves in Barbados and you never see your family again. However, I’m sure that’s just a myth…

And this, of course, can only be done because the body that tests every college has been officially elevated as the Education Tsar by each Caricom government. This is one of those areas where I am kind of doubtful about a private corporation being able to do it. No one but the government is allowed to exile all the professors each year and drown them in the caves if they fail. The only rule of survival is don’t go into the caves academia.

Accred Inc. starts with a significant capital investment. It devises this kind of a testing regime, offers it for a four-digit price per student, and conducts a lot of studies on how the test scores predict things like “will this person actually know how to do their job”. The testing regime is optimized for predictive power. Accred pays good money for people from the academical precariat to get sufficient expertise for the review process.

Blind review is utilized, and a lot of ~statistics~ is run on the answers to see if people can eg. recognize the characteristic style of some institutions, or the race or gender of the person answering, or any other irrelevant characteristic, because it can introduce biases. The aim is to have the answers be completely statistically indistinguishable when controlled for quality, so that the reviewers can’t cause conscious or unconscious favoritism. Reviewers are scored on their adherence to the consensus and those who deviate are dropped like uber drivers; reviewer scores might also be automatically adjusted so that if #526 tends to give everyone half a point less than the consensus the system gives it back if doing so improves the predictive power of the standardized testing.

Now, everyone who cares about work performance has an incentive to use Accred. Employers can hire equally skilled workers from less prestigious institutions; a cheap college + Accred fees saves students money compared to an expensive college; and expensive colleges need to prove they are worth the money.

Accred can use the fuckload of money it starts with to prove the superiority of objective standards in the media, it hires some really smart people who figure out how to spin it into a social justice issue (protip: it should hire me) of evil rich people pushing down competent poor people while simultaneously presenting it as “hey businesses, here’s a way to get free money by hiring smarter than the competitors” in a clever left-right pincer movement.

It clashes with the established education-industrial complex but thanks to ~free market magic~ it inevitably comes out as the winner because ~incentives are aligned~ (read: the VCs funding it pull some strings behind the scenes with their buddies) and it can use data to discredit any cheaper competitors who try to use less stringent standards for easier grading, and becomes a precarious monopoly which is profitable but must keep up with the demonstrable objectivity so that someone else doesn’t figure out how to do better than it.

One issue I can immediately perceive is that if Accred is cartelled out by prestigious professors and institutions and gets a reputation as a second-rate service for second-rate students, but even then a brutal focus on predictive power could show that even less prestigious graders can do a good job, and prestigious students could be offered money to defect and get tested and show that their credentials are less important than their score.

The biggest problem might be if the ~cathedral~ declares Accred evil and racist (~completely unlike~ the big name schools with their legacy admissions for incompetent whites and affirmative action aka. asian quotas) because black people score worse on average, and it gets regulated away like IQ testing. To pre-empt this, Accred needs to play the SJ card in its opening salvo, “we didn’t cause black people to test worse, the System is failing them but we developed a method to measure it so that now we can find out what actually helps with the achievement gap”, and put the establishment on defense. (protip: hire me to help with it) Furthermore, sufficiently reliable blind testing could reduce biases in employment when test scores can be given more weight than subjective evaluations. Self-taught people could test against those with degrees, and credentials could lose their importance in favor of measurable competence.

I totally see how it could possibly be done as a private enterprise, as long as it starts with enough money to afford the initial investment in proving its effectiveness with data. Anyone got Peter Thiel’s phone number?

3 months ago · tagged #win-win is my superpower #future precariat billionaire #ambitious trans girls · 113 notes · source: sinesalvatorem · .permalink

  1. almostcoralchaos reblogged this from sinesalvatorem
  2. semanticearth-community reblogged this from sinesalvatorem
  3. astaraeldarkrahblack reblogged this from wolffyluna
  4. wolffyluna reblogged this from sinesalvatorem
  5. davetheinverted reblogged this from inexacterminology
  6. keepsakewhales reblogged this from ilzolende and added:
    Actually, although standardized tests are often associated with bubble-in scantrons, they aren’t limited to those....
  7. towtowrage reblogged this from ilzolende
  8. ilzolende reblogged this from blastfarmer
  9. blastfarmer reblogged this from ilzolende and added:
    “Your budget is limited” is probably more a remark upon the state of funding for science than it is anything else....
  10. estelendur reblogged this from lethriloth and added:
    This sounds much better than having tests graded by minimum-wage peons who are punished for appreciating creativity in...
  11. zerofarad reblogged this from drethelin
  12. lethriloth reblogged this from sinesalvatorem
  13. socialjusticemunchkin reblogged this from sinesalvatorem and added:
    Accred Inc. starts with a significant capital investment. It devises this kind of a testing regime, offers it for a...
  14. umblrgumblr reblogged this from thetransintransgenic
  15. shacklesburst reblogged this from drethelin and added:
    Germany’s high school level final standardized tests sound a lot like the Caribbean system (or at least they did when I...
  16. drethelin reblogged this from plain-dealing-villain and added:
    I think one of my favorite things is people discovering that they’re using what they thought was a very specific phrase...
  17. plain-dealing-villain reblogged this from sinesalvatorem and added:
    Pretty sure the Carribean system wouldn’t survive scale-up, or contact with the high-stakes environment that is American...
  18. inexacterminology reblogged this from crowmeme
  19. sinesalvatorem reblogged this from blastfarmer and added:
    For college-level standardised tests in the Caribbean, there would be two separate papers per subject area, plus a...
  20. thetransintransgenic reblogged this from sinesalvatorem and added:
    Ah, see, in US English, “Standardized test” means “fill-in-the-bubbles scantron test”, and nothing else.It’s just one of...