Topics

❑ Introduction: Computer Science and Media Computation
1. ❑ Computer Science is about specifying process (recipes)
2. ❑ Why should you care about process?
  1. ❑ The Media Argument: If you ever want to say something that Adobe and Microsoft won't let you, you need to know something about programming
    1. ❑ All media are going digital
    2. ❑ Digital media are manipulated via software
    3. ❑ Programming (the creation of software) is a communications skill
3. ❑ Many fields are about exact specification of process
  1. ❑ Business, science
  2. ❑ Specification of process involves lots of aspects
    1. ❑ What do you name things?
    2. ❑ What are your units? How do you describe the things your process is working with?
    3. ❑ How do you specify what to do and when?
    4. ❑ How do you do this without driving yourself crazy writing down tedious detail over and over again?
    5. ❑ How can you do this efficiently?
4. ❑ KEY IDEA: Encoding
  1. ❑ Basically, computers only understand numbers
    1. ❑ Sequences of numbers from 0 to 255, to be exact
  2. ❑ But we can create standard definitions and agreements on how to interpret those numbers
    1. ❑ When you save an "A" in a file, your word processor actually stores the number 65. It agrees to INTERPRET that 65 as an A. (DEMO)
    2. ❑ Look at an A on the screen. It's actually a series of lit and unlit dots on the screen. We can represent a "graphical" A as a series of binary numbers that correspond to the dots on the screen.
  3. ❑ We can encode more complicated things by relying upon increasingly sophisticated encodings
    1. ❑ Complicated things like sounds, pictures, and movies (DEMO)
5. ❑ KEY IDEA: Naming
  1. ❑ Anything the computer knows about, we can associate with a name
  2. ❑ Its our own encoding -- we use it to establish our own conventions for how we want to think of things
  3. ❑ We can associate an encoding, like a sound, with a name, then manipulate the name like it's the original thing. (DEMO)
❑ Functions
1. ❑ We've already been asking the computer to do things for us
  1. ❑ Print
  2. ❑ Making and playing sounds
2. ❑ We can think of all of these doing things as functions
  1. ❑ Like the math idea: There's some input, something happens, and there's an output
  2. ❑ With computers, we don't always care about the output.
    1. ❑ Sometimes we care about what happens IN the box, like with Print or playing a sound
3. ❑ One of the things that a function can do for us in re-encode something or unencode it
  1. ❑ There are functions that move between different representations of basic data like letters and numbers (DEMO)
  2. ❑ But more interestingly, there are functions that allow us to take apart sounds and pictures into their component encodings
    1. ❑ What does that mean? To get there, we need to know a little about the standards and encodings for sounds and pictures.
    2. ❑ Some examples: Playing with the samples in sounds, playing with the pixels in pictures (DEMO)
❑ Arrays and loops
1. ❑ We talked about encodings and layers of encodings.
2. ❑ The simplest layer of an encoding is going from one number, to a series of numbers.
  1. ❑ KEY CS IDEA: We'll call that an array
3. ❑ We can create such series easily, [1 2 3], range(), etc. (DEMO)
4. ❑ More importantly, sounds and pictures can be understood as an array of encodings (numbers), and we can manipulate these.
5. ❑ How do we manipulate arrays? Typically with a loop
  1. ❑ We can do something to each element of an array with FOR X in ARRAY
6. ❑ Example: Let's do something to each "sample" of a sound (DEMO)
  1. ❑ Multiply it by 2
  2. ❑ Multiply it by 0.5
❑ Sound
1. ❑ Let's talk a bit about sound, using the MediaTools for examples, so that we can figure out what we're doing with it.
2. ❑ Sound is a wave
  1. ❑ Physically, it's molecules moving around in the air -- hitting one another, and then getting bounced away. That creates pressure and less-pressure.
  2. ❑ We can see in an oscilloscope the increasing and decreasing pressure (DEMO)
    1. ❑ Increasing pressure is positive
    2. ❑ Decreasing pressure is negative
    3. ❑ Zero pressure is, well, zero
  3. ❑ We can associate numbers with the amount of pressure (actually, it's the voltage coming from the microphone) (DEMO)
    1. ❑ We can see that LOUDER sounds create a greater variation in the pressure: Difference from top to bottom gets large
    2. ❑ softer sounds stay closer to zero
  4. ❑ Each of those numbers is called a sample
3. ❑ Recording sound well takes LOTS of samples
  1. ❑ We can hear (most of us -- less so for us older folk) between 7 Khz and 22 Khz
    1. ❑ That's 7000 ups-and-downs (one cycle) per second, and 22,000 cycles per second
  2. ❑ There's a mathematical result that says that if you want to capture everything that occurs in a sound, you have to capture at TWICE the frequency of the highest sound you want to capture
    1. ❑ 2 * 22 Khz = 44 Khz
    2. ❑ This means that we have to capture 44,000 samples (numbers) PER SECOND to get everything in a recording that we might hear.
    3. ❑ That's the rate at which a CD is recorded. 44,000 numbers per second. In an array (a sequence)
4. ❑ What were we doing when we multiplied or divided the samples? Increased or decreased the range => Increased or decreased the volume (DEMO)
❑ Developing a mental model of the program: Debugging
1. ❑ Computers only do what you tell them to do
  1. ❑ Computers only do what you tell them to do
    1. ❑ Computers only do what you tell them to do
2. ❑ The tricky part is figuring out exactly what you told them to do
  1. ❑ There are two things that we're concerned with
    1. ❑ The "Flow of execution" -- what happens first, then second, then whatever
    2. ❑ The "Flow of data" -- what variables got what when
3. ❑ Both parts can be studied by adding Print statements
  1. ❑ Print statements can be signposts telling you where the code is at and when (DEMO)
    1. ❑ Is your code running slowly? Where IS your execution? What's the computer doing NOW? A well-placed Print statement can show you that
  2. ❑ Print statements can also show you what the (invisible) variables values are
4. ❑ So far, we've only been dealing with linear flows of execution and iterative flows of execution (looping). Conditional is still to come, and then it gets trickier to track the flow
  1. ❑ Using show_vars for debugging trickier situations (DEMO)
❑ Sound, Part 2
1. ❑ We can use loops for more than just walking all the values of an array: We can also use them for generating values
  1. ❑ Think about what happens when a sound is played
    1. ❑ The samples are sent to the speaker, one at a time, at the same rate as they were recorded
  2. ❑ Consider what would happen if we skipped every other sample when we played it back.
    1. ❑ We'd double the frequency and half the time (DEMO)
  3. ❑ What about if we skipped 0.5 samples each time (i.e., sent a sample twice)
    1. ❑ We'd half the frequency and double the time (DEMO)
  4. ❑ Can we play with the frequency by changing the playback rate? Can we try this for a range of numbers?
    1. ❑ Sure, but isn't it tedious to type in all these examples?
    2. ❑ For x in [0.1 0.2 0.5 1.0 1.5 2.0], play sound at freq x (DEMO)
  5. ❑ What happens if we add samples? We can create REVERB! (DEMO)
    1. ❑ We'll need two sounds, and we'll add from one to the other
    2. ❑ Now we'll need TWO loops
      1. ❑ One will track where we are in the source sample
      2. ❑ The other one will handle the fading out of the sound over time
❑ Images/Pictures
1. ❑ KEY CS IDEA: A linear sequence of values is ONE way to think about data. Another common way is with a table
  1. ❑ Examples of tables: From newspapers, from textbooks, from lots of places
  2. ❑ Some recipes need more than a series
2. ❑ A picture is actually a table of pixels
  1. ❑ Remember a sound was an array of samples, where each sample was just a number
  2. ❑ A picture is a table (not just a sequence) of a more sophisticated encoding.
    1. ❑ Not one number, but three.
    2. ❑ A value for the amount of redness (0 to 255)
    3. ❑ A value for the amount of greenness, of blueness, and of "transparency" (called alpha), all 0 to 255
  3. ❑ KEY CS IDEA: We can use representations and encodings within one another
  4. ❑ This is how it really works. Go put a magnifying class on a TV, or a monitor, or even an LCD. The screen is made up of "dots" (picture elements => pixels), and each dot has smaller dots corresponding to different colors.
3. ❑ We can load pictures, see their pixel values, set their pixel values (DEMO)
❑ Manipulating images: Changing ARGB values, Filtering, functions
1. ❑ Pictures are made of pixels that we can change
2. ❑ We can walk through an image and change all the reds to less red, or more red (DEMO)
3. ❑ Same for green, blue, or alpha (DEMO)
4. ❑ KEY CS IDEA We can make a function and apply it to a BUNCH of data
  1. ❑ Make up a function to do some kind of filtering (changing of pixel values) and use a loop to apply the function to each pixel in a picture
❑ Filtering Part 2: Using more sophisticated functions. Conditionals
1. ❑ By just increasing/decreasing pixel values, we're doing simple PhotoShop-style filtering, but that's pretty simplistic.
  1. ❑ It's called a linear function
2. ❑ Often, you want to make choices about pixels and treat them differently
  1. ❑ For example: Everything that's mostly red (say, over 200), decrease the red. If the redness is less than 200, leave it alone
  2. ❑ This is a form of thresholding: There's a threshold value that determines what you do.
3. ❑ KEY CS IDEA Computers don't have to do just one thing after another, nor just loop. They can also make choices
  1. ❑ But real limited choices: They only understand number, they only understand numbers, and they only understand numbers
4. ❑ We can use an If-Then which is a Test.
  1. ❑ IF this is true, Then the computer must, Must, MUST do the Then
  2. ❑ No choices about it. "Test this, computer. And if it's true, DO IT!"
5. ❑ Change the redness, if needed (DEMO)
6. ❑ Creating more kinds of threshold functions (DEMO)
❑ Working with a portion of an image: Masks and looping
1. ❑ Oftentimes you don't want to do something to an entire image. Instead, you want to apply some filter to just part of an image
  1. ❑ For example, turn the lady's hat red; blur only the one character; blur everything BUT the one character; make the box eerily-transparent
2. ❑ How do we do that? Easiest way: change the limits on your loops
  1. ❑ Don't go from 1 to the width and 1 to the height.
  2. ❑ Instead, just go from 10 to 250 and 5 to 100 (for example) DEMO
3. ❑ But that approach can only handle a rectangle. Most areas you want to manipulate are more complicated
  1. ❑ The answer: Compute a mask
  2. ❑ This is an unusual thing. It's a "picture" (of a sort) with data in it that does NOT correspond to pixels. Instead we just store 0's and 1's in it.
    1. ❑ KEY CS IDEA: Creating some data just to make the "recipe" easier
    2. ❑ 1's mean "This is part of the picture that we want to process"
    3. ❑ 0's mean "This is part of the picture that we do NOT want to process"
  3. ❑ You end up going through all the pixels TWICE
    1. ❑ First time, with a conditional, to decide whether you want to process that part of the picture. If so, put a 1 in the corresponding parts of the mask, and a 0 in all the others
    2. ❑ Now, go through all the pixels, and where ever there's a 1, apply the filter/function. Otherwise, don't.
❑ Drawing on an image: Adding lines, circles, text, and other elements
❑ Developing a mental model of the program: Debugging conditionals
1. ❑ How do you figure out what happened where? Especially with sequential, iterative, and conditional computation?
2. ❑ KEY CS IDEA: Play computer! Trace the program and do as it would do
  1. ❑ Print statements to find out where it went and what the data values are is KEY
  2. ❑ Important to look at the input: What's really going on?
    1. ❑ In real software development, you work at figuring out the input that breaks a program
    2. ❑ One way of finding these is to try the boundary conditions: What happens at 0? At 255?
    3. ❑ Then you make it work for that input too
❑ Files
1. ❑ Media are mostly stored in files today. If you want to manipulate lots of media, you need to manipulate files.
2. ❑ There are more ways of encoding data than just arrays and tables
  1. ❑ KEY CS IDEA Frequently, you want a hierarchy, which is represented as a tree
  2. ❑ Think about outlines, about the structure of dictionaries and encylopediae
  3. ❑ You've seen this already in file structures. Directories are nodes in a tree. Files in the directory (a sequence within the node) are peers or children of the directory
  4. ❑ We can use the directory tree to find things, move things around
3. ❑ Remember how pixels are inside pictures -- a sophisticated encoding inside of another encoding? Files are similar
  1. ❑ Files have contents, creation dates, names -- all inside of a linear list inside a directory, which is part of a tree
  2. ❑ It's data representations and encodings all the way up and down!
4. ❑ We can write programs to manipulate all the sounds or images in a directory (DEMO)
  1. ❑ We use loops to walk the sequence in a directory
❑ Writing Utility Functions: Moving/manipulating your files
1. ❑ We can use the other ideas we've introduced, like conditionals
  1. ❑ For example, process only the sounds whose names end in ".wav" (DEMO)
    1. ❑ We'll have to do a little string manipulation here
    2. ❑ A string is a linear encoding of characters
  2. ❑ Or, process only the sounds whose modification date is today (DEMO)
    1. ❑ Dates are another encoding on numbers
2. ❑ We can also copy files from one place to another
  1. ❑ But how we do it depends on the encoding in the file
    1. ❑ You can always read and write the numbers. This is called a binary copying (for reasons we'll see later)
    2. ❑ We could also read and write the letters, the strings. This is assuming the data is text
    3. ❑ We can use a loop to get all data across
    4. ❑ KEY CS IDEA: But a loop that doesn't count. Instead, it tests if we're at the end of the file. A WHILE
❑ Video: A series of pictures/frames
1. ❑ A Video is a series of images called frames
2. ❑ We can process them JUST like we did images, but we have to do it for every frame
3. ❑ Easiest way: Convert your video to frames, manipulate the frames, reassemble them into a movie
  1. ❑ Demonstration with MediaTools (DEMO)
4. ❑ Now, apply some processing to those frames, by combining what we know about processing files with what we know about manipulating images (DEMO)
❑ Filtering a range of pictures
1. ❑ What if we want to do something to just SOME of the frames? Same techniques as processing just SOME of the pixels
  1. ❑ Insert a blue balloon in SOME of the frames of a picture (DEMO)
2. ❑ We can also do masking
  1. ❑ How the weatherman works! Background subtraction and "bluescreening" (DEMO)
❑ "WHY IS THIS TAKING SO LONG?!?"
1. ❑ Why is video processing taking so long? Is it just the speed of my processor?
2. ❑ KEY CS IDEA: The "order" (Big Oh) of an algorithm
  1. ❑ The more loops you have, the more basic operations you do
  2. ❑ Array processing is order n, O(n)
  3. ❑ Table/matrix processing is O(n*m)
  4. ❑ If you have f frames in a video, that's O(n*m*f)
  5. ❑ What if you compute a mask for each frame THEN process it? O(2*n*m*f)
3. ❑ KEY CS IDEA: Moore's Law: Every 18 months, the processor speed doubles for the same cost
  1. ❑ But that only cuts the time cost in HALF
  2. ❑ That's great, but it doesn't make processing a movie like processing an array
4. ❑ How do we make things faster? Fixing how we specify the recipe/process
  1. ❑ An example: Finding things in an array
    1. ❑ Just searching one-after-the-other: O(n)
    2. ❑ What if they're in a particular order? We can use a binary search O(log n)
      1. ❑ Example: How to search a dictionary or phone book efficiently (DEMO)
    3. ❑ How do we get things in order? Sorting.
      1. ❑ Worst we can do O(n*n)
        
        ❑ Compare everything to everything
      2. ❑ Best we can do is probably O(n log n) (Not planning to give more than an intuitive sense here)
5. ❑ Guess what: Some things CAN'T be made faster!
  1. ❑ Imagine an optimal arrangement of sounds in a composition/synthesis. You have to check EVERY combination.
  2. ❑ Let's say that you have 60 sounds you want to arrange and any order is possible, but you want to figure out the best one
    1. ❑ Basically, if you have to try every combination of n things, there are 2 ^ n combinations (can demo this pretty easily)
    2. ❑ O(2 ^ 60) = 11,52,921,504,606,846,976
    3. ❑ Imagine that you have a 1 Ghz computer (1 billion basic operations per second) -- a top of the line processor today
    4. ❑ It'll take you 1152921504.606847 seconds to optimize that data
      1. ❑ That's 19,215,358.41011412 minutes
      2. ❑ That's 800,639.933754755 days
      3. ❑ That's 2,193 years
      4. ❑ With Moore's law, in two years, you can do that in only 1,000 years!
      5. ❑ And 60 sounds is a SMALL amount -- most songs have many more notes than that
  3. ❑ Can we do better? Maybe -- can you be satisfied with less than perfect? Can we be smarter than checking EVERY combination? THAT'S PART OF WHAT COMPUTER SCIENTISTS DO!
❑ Text as a Media Type
1. ❑ KEY CS IDEA: Text itself is a media type that computers are good at manipulating
2. ❑ We've already been doing some string manipulation.
3. ❑ Imagine that you've got a file with RGB values specified as TEXT. Can we turn that into a picture? DEMO
4. ❑ Can we go from a picture and generate those RGB values? DEMO
5. ❑ KEY CS IDEA: This is interpretation -- we're moving between encodings, and using an encoding to tell the computer what to do.
6. ❑ If text is language, we can process it, still, but it's less well-formed so harder to process
  1. ❑ A demo or two goes here
❑ Making other programs do the work
1. ❑ Writing fast, smart algorithms is a lot of work.
  1. ❑ Let other people do it!
  2. ❑ We can use what they produce!
2. ❑ CS CONCEPT: We can write programs to control other programs
  1. ❑ Sometimes called scripting
3. ❑ Not all programs are scriptable! Photoshop is not. GIMP, a free, open-source, cross-platform version of PhotoShop is
4. ❑ Multiple ways of scripting
  1. ❑ Sometimes, can just call functions within your own language, and inter-application communication handles it
  2. ❑ Sometimes, you write a program that creates files that the other program understands
    1. ❑ Command files
    2. ❑ Data files
5. ❑ Demonstrate both techniques for controlling GIMP
❑ Graphing data
1. ❑ What data/information visualizers do: Take some data, turn it into a picture
2. ❑ We can do that ourselves, using what we know about graphics, reading files, and interpreting text
  1. ❑ Building a simple scatterplot, say with stock quotes or other data from the Web DEMO
3. ❑ But building graphs is hard -- let's make somebody else do it!
  1. ❑ We can get data ready for Excel
    1. ❑ Demo creating a tab-delimited file of data
  2. ❑ Or we can use a sophisticated plotting package
❑ Graphing data with GNUPlot
1. ❑ Use what we know about using other programs and make GNUPlot do it!
2. ❑ Get data ready for GNUPlot
❑ "Can't we do this any easier?": Functional Decomposition
1. ❑ Writing lines and lines and lines of code is tedious and error prone. How do we make it easier?
2. ❑ Wherever possible, create a function to do it!
  1. ❑ Remember how we created a function with our filter to apply to our images?
  2. ❑ We can use this idea more generally
❑ "Can't we do this any easier?": Recursion
1. ❑ Some problems are more easily solved by doing something other than straight sequential, iterative, and conditional computation
2. ❑ KEY CS IDEA: Recursion is functional programming to the max! A function calling itself!
  1. ❑ Use all the Brian Harvey ways of thinking about recursion here
❑ "Can't we do this any easier?": Functional programming
1. ❑ We can take functions further
2. ❑ Remember when we said in Week 1 that ANYTHING can have a name? So can functions!
3. ❑ Remember how functions took inputs? Functions can also be inputs to other functions!
4. ❑ We can apply function to data, rather than pass data to functions
  1. ❑ Reduce, Map, Apply, Filter/Select
❑ "Can't we do this any easier?": Objects
1. ❑ Pixels and files, and pictures and directories, are more than just sequences of numbers
2. ❑ How do we represent these kinds of related encodings?
3. ❑ Functions that manipulate these complex encodings probably differ depending on the data we're manipulating
  1. ❑ Sometimes we may want to re-use the same names
  2. ❑ Sometimes we don't even want to use functions -- instead, we want to touch the numbers (like red value) directly!
4. ❑ KEY CS IDEA: An Object encapsulates (combines) data and behavior into one encoding that we can manipulate more easily
  1. ❑ Pixels as objects
  2. ❑ Files as objects
5. ❑ How we make objects (DEMO)
❑ Re-visiting media manipulation as functional and object-oriented programming
1. ❑ Re-do some of the earlier examples, but in terms of functions
2. ❑ Re-do some of the earlier examples, but in terms of objects
❑ Languages and Representations for Recipes: It's much of what computer scientists do
1. ❑ Jython isn't the only language that computer scientists use.
2. ❑ Other languages allow us to represent data or process (recipes) better for some kinds of problems
3. ❑ Examples: No more than 2-3 screens each, just to give a flavor, the "sound" of the language
  1. ❑ Lists and functions and functional programming in Lisp
  2. ❑ Manipulating pixels and sounds in Smalltalk
  3. ❑ Manipulating arrays and matrices in MATLAB
4. ❑ Much of the world uses Java these days: Let's talk about what Java looks like
❑ Introduction to Java
1. ❑ Using Java; Code goes in files, we "compile" the files, we execute the "object" code
2. ❑ All of Java is objects
  1. ❑ How classes are defined in Java
3. ❑ All data in Java is "typed"
  1. ❑ Tell it the encoding!
4. ❑ Walking a Java file -- what it looks like
❑ Introduction to Java Media Manipulation
1. ❑ We can call functions (called methods) in Java to do all that we did before
2. ❑ Picture and Image manipulation in Java EXAMPLES
❑ Spare slot and Final Exam Review